{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "EheA5_j_cEwc"
      },
      "source": [
        "##### Copyright 2019 Google LLC.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "YCriMWd-pRTP"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\"); { display-mode: \"form\" }\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "OvRwFTkqcp1e"
      },
      "source": [
        "# Introduction to TensorFlow Part 3 - Advanced Tensor Manipulation\n",
        "\n",
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/google/tf-quant-finance/blob/master/tf_quant_finance/examples/jupyter_notebooks/Introduction_to_TensorFlow.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/google/tf-quant-finance/blob/master/tf_quant_finance/examples/jupyter_notebooks/Introduction_to_TensorFlow.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "nPX9m7Q_w_8p"
      },
      "outputs": [],
      "source": [
        "#@title Install and import Libraries for this colab. RUN ME FIRST!\n",
        "!pip install matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf\n",
        "\n",
        "tf.disable_eager_execution()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "obZHjPQScSqd"
      },
      "source": [
        "# What this notebook covers\n",
        "\n",
        "This notebook carries on from [part 2](https://colab.research.google.com/github/google/tf-quant-finance/blob/master/tf_quant_finance/examples/jupyter_notebooks/Introduction_to_TensorFlow_Part_2_-_Debugging_and_Control_Flow.ipynb\n",
        "), and covers various advanced ways of manipulating tensors, including\n",
        "*   Scatter/Gather\n",
        "*   Sparse Tensors\n",
        "*   Various functional ops\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pVLqCYzuDhfT"
      },
      "source": [
        "# Scatter / Gather"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_md9nF-EbCcn"
      },
      "source": [
        "## tf.gather_nd\n",
        "\n",
        "[Full documentation](https://www.tensorflow.org/api_docs/python/tf/gather_nd)\n",
        "\n",
        "This operation allows you to take a multi-dimensional tensor and extract a list of subsets of data from it, according to a list of indices."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        },
        "colab_type": "code",
        "executionInfo": {
          "elapsed": 631,
          "status": "ok",
          "timestamp": 1568638253674,
          "user": {
            "displayName": "Mark Charsley",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mAuhiLGs9-CmN1BMtie2tWn3oucS__w2c4gNC6yBg=s64",
            "userId": "18064761308611293318"
          },
          "user_tz": -60
        },
        "id": "z7nnDntT32V0",
        "outputId": "80c44424-c641-4504-c7d9-ae6ad71d8b11"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Looking up [[1, 1, 1]] gives us\n",
            "[222]\n",
            "\n",
            "Looking up [[1, 1, 1], [0, 0, 0], [0, 0, 1]] gives us\n",
            "[222 111 112]\n",
            "\n",
            "Looking up [[0, 0]] gives us\n",
            "[[111 112 113]]\n",
            "\n",
            "Looking up [[1]] gives us\n",
            "[[[211 212 213]\n",
            "  [221 222 223]\n",
            "  [231 232 233]]]\n"
          ]
        }
      ],
      "source": [
        "source = tf.constant([[[111,112,113], [121,122,123], [131,132,133]], \n",
        "                      [[211,212,213], [221,222,223], [231,232,233]]])\n",
        "\n",
        "with tf.Session() as sess:\n",
        "  # if we specify all values for all of source's dimensions, then we get a \n",
        "  # single value\n",
        "  indices = [[1,1,1]]\n",
        "  print(\"Looking up %s gives us\\n%s\" %(\n",
        "      indices, sess.run(tf.gather_nd(source, indices))))\n",
        "\n",
        "  # we can look up multiple sets of indices\n",
        "  indices = [[1,1,1], [0,0,0], [0,0,1]]\n",
        "  print(\"\\nLooking up %s gives us\\n%s\" %(\n",
        "      indices, sess.run(tf.gather_nd(source, indices))))\n",
        "\n",
        "  # if we don't specify values for all of source's dimensions, then we get \n",
        "  # results of larger shape\n",
        "  indices = [[0,0]]\n",
        "  print(\"\\nLooking up %s gives us\\n%s\" %(\n",
        "      indices, sess.run(tf.gather_nd(source, indices))))  \n",
        "  indices = [[1]]\n",
        "  print(\"\\nLooking up %s gives us\\n%s\" %(\n",
        "      indices, sess.run(tf.gather_nd(source, indices))))    "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "9B9cza9zD4Iw"
      },
      "source": [
        "The indices can easily be generated with tf.where_v2:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "colab_type": "code",
        "executionInfo": {
          "elapsed": 436,
          "status": "ok",
          "timestamp": 1568385897609,
          "user": {
            "displayName": "Mark Charsley",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mAuhiLGs9-CmN1BMtie2tWn3oucS__w2c4gNC6yBg=s64",
            "userId": "18064761308611293318"
          },
          "user_tz": -60
        },
        "id": "npOjtvGYECW1",
        "outputId": "f2a94d9d-5420-4cb4-b30c-787660c35a4c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[111 123 132 213 222 231]\n"
          ]
        }
      ],
      "source": [
        "source = tf.constant([[[111,112,113], [121,122,123], [131,132,133]], \n",
        "                      [[211,212,213], [221,222,223], [231,232,233]]])\n",
        "values_divisible_by_three = tf.gather_nd(\n",
        "    source, tf.where_v2(tf.equal(0, source % 3)))\n",
        "\n",
        "with tf.Session() as sess:\n",
        "  print(sess.run(values_divisible_by_three))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "EfNajZEZG3sd"
      },
      "source": [
        "##tf.scatter_nd\n",
        "\n",
        "[Full documentation](https://www.tensorflow.org/api_docs/python/tf/scatter_nd)\n",
        "\n",
        "scatter_nd creates a zero-initialised tensor of a given shape, and then writes a series of specified values at specified positions in that tensor.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 756
        },
        "colab_type": "code",
        "executionInfo": {
          "elapsed": 656,
          "status": "ok",
          "timestamp": 1568708441852,
          "user": {
            "displayName": "Mark Charsley",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mAuhiLGs9-CmN1BMtie2tWn3oucS__w2c4gNC6yBg=s64",
            "userId": "18064761308611293318"
          },
          "user_tz": -60
        },
        "id": "9dKH_Fg4H4R4",
        "outputId": "4210e38c-5e5d-4299-b471-f7cdd5b31e33"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Scattering single value:\n",
            "[[[0 0 0]\n",
            "  [0 0 1]\n",
            "  [0 0 0]]\n",
            "\n",
            " [[0 0 0]\n",
            "  [0 0 0]\n",
            "  [0 0 0]]]\n",
            "\n",
            "Scattering multiple values:\n",
            "[[[1 0 0]\n",
            "  [0 2 0]\n",
            "  [0 0 3]]\n",
            "\n",
            " [[0 0 0]\n",
            "  [0 0 0]\n",
            "  [0 0 0]]]\n",
            "\n",
            "Scattering entire rows:\n",
            "[[[1 2 3]\n",
            "  [4 5 6]\n",
            "  [0 0 0]]\n",
            "\n",
            " [[0 0 0]\n",
            "  [0 0 0]\n",
            "  [0 0 0]]]\n",
            "\n",
            "Scattering entire matrix:\n",
            "[[[1 2 3]\n",
            "  [4 5 6]\n",
            "  [7 8 9]]\n",
            "\n",
            " [[0 0 0]\n",
            "  [0 0 0]\n",
            "  [0 0 0]]]\n",
            "\n",
            "Scattering single value multiple times:\n",
            "[[[6 0 0]\n",
            "  [0 0 0]\n",
            "  [0 0 0]]\n",
            "\n",
            " [[0 0 0]\n",
            "  [0 0 0]\n",
            "  [0 0 0]]]\n"
          ]
        }
      ],
      "source": [
        "source = tf.constant([[[111,112,113], [121,122,123], [131,132,133]], \n",
        "                      [[211,212,213], [221,222,223], [231,232,233]]])\n",
        "\n",
        "with tf.Session() as sess:\n",
        "  print(\"Scattering single value:\\n%s\"%sess.run(\n",
        "      tf.scatter_nd(\n",
        "          indices = [[0,1,2]], \n",
        "          updates = [1], \n",
        "          shape = [2,3,3])))\n",
        "\n",
        "  print(\"\\nScattering multiple values:\\n%s\"%sess.run(\n",
        "      tf.scatter_nd(\n",
        "          indices = [[0,0,0], [0,1,1], [0,2,2]], \n",
        "          updates = [1,2,3], \n",
        "          shape = [2,3,3])))\n",
        "\n",
        "  # You can reduce the dimensions of indices and increase the dimensions of \n",
        "  # updates\n",
        "  print(\"\\nScattering entire rows:\\n%s\"%sess.run(\n",
        "      tf.scatter_nd(\n",
        "          indices = [[0,0], [0,1]], \n",
        "          updates = [[1,2,3], [4,5,6]], \n",
        "          shape = [2,3,3])))\n",
        "  \n",
        "  print(\"\\nScattering entire matrix:\\n%s\"%sess.run(\n",
        "      tf.scatter_nd(\n",
        "          indices = [[0]], \n",
        "          updates = [[[1,2,3], [4,5,6], [7,8,9]]], \n",
        "          shape = [2,3,3])))\n",
        "  \n",
        "  # Note that if indices contains duplicate or overlapping values, then the \n",
        "  # clashing updates will be added together (in an indeterminate order, which \n",
        "  # may result in non-deterministic output in the case of multiple floating \n",
        "  # point values of wildly different sizes).\n",
        "  print(\"\\nScattering single value multiple times:\\n%s\"%sess.run(\n",
        "      tf.scatter_nd(\n",
        "          indices = [[0,0,0], [0,0,0], [0,0,0]], \n",
        "          updates = [1,2,3], \n",
        "          shape = [2,3,3])))\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "kYiOe3ZsT05b"
      },
      "source": [
        "### Gather then Scatter\n",
        "In some cases, you will want scatter_nd to act as a \"setter\" to gather_nd's \"getter\": i.e. you have a tensor, you extract a subset of values that meet a certain criteria using gather_nd, you calculate new values for that subset, and then create a new tensor based on the original that replaces the elements that met the criteria with the new values.\n",
        "\n",
        "There are two approaches you can take here, the first is to use where_v2:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 134
        },
        "colab_type": "code",
        "executionInfo": {
          "elapsed": 441,
          "status": "ok",
          "timestamp": 1568709877819,
          "user": {
            "displayName": "Mark Charsley",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mAuhiLGs9-CmN1BMtie2tWn3oucS__w2c4gNC6yBg=s64",
            "userId": "18064761308611293318"
          },
          "user_tz": -60
        },
        "id": "gbpQx-xrU707",
        "outputId": "f5046ba6-5ec1-4940-de01-9d79496d178b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[[[ 11 112 113]\n",
            "  [121 122  23]\n",
            "  [131  32 133]]\n",
            "\n",
            " [[211 212  13]\n",
            "  [221  22 223]\n",
            "  [ 31 232 233]]]\n"
          ]
        }
      ],
      "source": [
        "source = tf.constant([[[111,112,113], [121,122,123], [131,132,133]], \n",
        "                      [[211,212,213], [221,222,223], [231,232,233]]])\n",
        "# create a boolean tensor the same shape as source\n",
        "is_divisible_by_three = tf.equal(0, source % 3)\n",
        "# create a list of indices where is_divisible_by_three is true\n",
        "indices = tf.where_v2(is_divisible_by_three)\n",
        "# extract a list of values that need updating\n",
        "values_divisible_by_three = tf.gather_nd(source, indices)\n",
        "# perform a really expensive operation on those values\n",
        "new_values = values_divisible_by_three % 100\n",
        "# convert the list of new values back into a tensor the same shape as source\n",
        "# we need to force the type of shape to match that of indices\n",
        "shape = tf.shape(source, out_type=tf.dtypes.int64)\n",
        "new_values_or_zeros = tf.scatter_nd(indices, new_values, shape)\n",
        "# merge the new values into the original\n",
        "new_tensor = tf.where_v2(is_divisible_by_three, new_values_or_zeros, source)\n",
        "\n",
        "with tf.Session() as sess:\n",
        "  print (sess.run(new_tensor))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "OIMX9Vf-Zmdf"
      },
      "source": [
        "The second approach is to subtract the original value from the updated value, and then you can just add the two tensors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 134
        },
        "colab_type": "code",
        "executionInfo": {
          "elapsed": 443,
          "status": "ok",
          "timestamp": 1568710356563,
          "user": {
            "displayName": "Mark Charsley",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mAuhiLGs9-CmN1BMtie2tWn3oucS__w2c4gNC6yBg=s64",
            "userId": "18064761308611293318"
          },
          "user_tz": -60
        },
        "id": "tMwh5kgAZ0aI",
        "outputId": "04aabaf3-8dfa-4dbd-e6ff-867baac23847"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[[[ 11 112 113]\n",
            "  [121 122  23]\n",
            "  [131  32 133]]\n",
            "\n",
            " [[211 212  13]\n",
            "  [221  22 223]\n",
            "  [ 31 232 233]]]\n"
          ]
        }
      ],
      "source": [
        "source = tf.constant([[[111,112,113], [121,122,123], [131,132,133]], \n",
        "                      [[211,212,213], [221,222,223], [231,232,233]]])\n",
        "# create a list of indices where is_divisible_by_three is true (we no longer \n",
        "# need the to keep a reference to the result of tf.equal)\n",
        "indices = tf.where_v2(tf.equal(0, source % 3))\n",
        "# extract a list of values that need updating\n",
        "values_divisible_by_three = tf.gather_nd(source, indices)\n",
        "# perform a really expensive operation on those values to get the value we want\n",
        "new_values = values_divisible_by_three % 100\n",
        "new_values_minus_original = new_values - values_divisible_by_three\n",
        "# convert the list of new values back into a tensor the same shape as source\n",
        "# we need to force the type of shape to match that of indices\n",
        "shape = tf.shape(source, out_type=tf.dtypes.int64)\n",
        "new_values_or_zeros = tf.scatter_nd(indices, new_values_minus_original, shape)\n",
        "# now add the two tensors. For values that didn't meet the criteria this just \n",
        "# add 0 to the original value. For the values that did meet the criteria this \n",
        "# will add original_value + (value_we_want - original value)\n",
        "new_tensor = new_values_or_zeros + source\n",
        "\n",
        "with tf.Session() as sess:\n",
        "  print (sess.run(new_tensor))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "W6-o_3hTeCb1"
      },
      "source": [
        "The addition approach is fractionally more efficient, and doesn't require the boolean tensor, but can hit accuracy problems in floating point."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pnyntx09GpYW"
      },
      "source": [
        "### tf.scatter_nd_add/sub/update\n",
        "\n",
        "There are several other scatter_nd functions in tensorflow which are superficially similar to ```tf.scatter_nd```. They work using [Variables](https://www.tensorflow.org/guide/variables) which can have significant scalability/contention issues with larger graphs, especially when executing across distributed machines/hardware."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "FOqSOkTAH9No"
      },
      "source": [
        "## Exercise: Mandlebrot set\n",
        "\n",
        "Lets revisit the Mandlebrot set from the previous training course. In that solution, we ran the z=z*z+c calculation for all co-ordinates, even the ones whose magnitude had already gone over 2.\n",
        "\n",
        "For the purpose of this exercise, we will pretend that the complex calculation is very expensive and that we should eliminate the calculation where possible. In actual fact, the calculation is utterly trivial and swamped by the cost of the gather/scatter operations, but the same methods can be used in situations rather more expensive than a complex add and multiply"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "DZZxF2l0it9-"
      },
      "outputs": [],
      "source": [
        "MAX_ITERATIONS = 64\n",
        "NUM_PIXELS = 512\n",
        "\n",
        "def generate_grid(nX, nY, bottom_left=(-1.0, -1.0), top_right=(1.0, 1.0)):\n",
        "  \"\"\"Generates a complex matrix of shape [nX, nY].\n",
        "  \n",
        "  Generates an evenly spaced grid of complex numbers spanning the rectangle \n",
        "  between the supplied diagonal points. \n",
        "  \n",
        "  Args:\n",
        "    nX: A positive integer. The number of points in the horizontal direction.\n",
        "    nY: A positive integer. The number of points in the vertical direction.\n",
        "    bottom_left: The coordinates of the bottom left corner of the rectangle to\n",
        "      cover.\n",
        "    top_right: The coordinates of the top right corner of the rectangle to\n",
        "      cover.\n",
        "\n",
        "  Returns:\n",
        "    A constant tensor of type complex64 and shape [nX, nY].\n",
        "  \"\"\"\n",
        "  x = tf.linspace(bottom_left[0], top_right[0], nX)\n",
        "  y = tf.linspace(bottom_left[1], top_right[1], nY)\n",
        "  real, imag = tf.meshgrid(x, y)\n",
        "  return tf.cast(tf.complex(real, imag), tf.complex128)\n",
        "\n",
        "c_values = generate_grid(NUM_PIXELS, NUM_PIXELS)\n",
        "initial_Z_values = tf.zeros_like(c_values, dtype=tf.complex128)\n",
        "initial_diverged_after = tf.ones_like(c_values, dtype=tf.int32) * MAX_ITERATIONS\n",
        "\n",
        "# You need to put the various values you want to change inside the loop here\n",
        "loop_vars = (0, initial_Z_values, initial_diverged_after)\n",
        "\n",
        "# this needs to take the same number of arguments as loop_vars contains and\n",
        "# return a tuple of equal size with the next iteration's values\n",
        "def body(iteration_count, Z_values, diverged_after):\n",
        "  # a matrix of bools showing all the co-ordinatesthat haven't diverged yet\n",
        "  not_diverged = tf.equal(diverged_after, MAX_ITERATIONS)\n",
        "  # a list of the indices in not_diverged that are true\n",
        "  not_diverged_indices = tf.where_v2(not_diverged)\n",
        "\n",
        "  # you now need to gather just the Z and c values covered by \n",
        "  # not_diverged_indices, calculate the new Z values, and then scatter the \n",
        "  # values back into a new Z_values matrix to pass to the next iteration.\n",
        "\n",
        "  new_Z_values = # TODO\n",
        "\n",
        "  # And now we're back to the original code\n",
        "  has_diverged = tf.abs(new_Z_values) \u003e 2.0\n",
        "  new_diverged_after = tf.minimum(diverged_after, tf.where_v2(\n",
        "      has_diverged, iteration_count, MAX_ITERATIONS))\n",
        "\n",
        "  return (iteration_count+1, new_Z_values, new_diverged_after)\n",
        "\n",
        "# this just needs to take the same number of arguments as loop_vars contains and\n",
        "# return true (we'll use maximum_iterations to exit the loop)\n",
        "def cond(iteration_count, Z_values, diverged_after):\n",
        "  return True\n",
        "\n",
        "loop = tf.compat.v1.while_loop(\n",
        "    loop_vars=loop_vars, \n",
        "    body = body, \n",
        "    cond = cond, \n",
        "    maximum_iterations=MAX_ITERATIONS)\n",
        "\n",
        "with tf.Session() as sess:\n",
        "  results = sess.run(loop)\n",
        "\n",
        "## extract the final value of diverged_after from the tuple\n",
        "final_diverged_after = results[-1]\n",
        "plt.matshow(final_diverged_after)\n",
        "pass"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        },
        "colab_type": "code",
        "executionInfo": {
          "elapsed": 2703,
          "status": "ok",
          "timestamp": 1568628827392,
          "user": {
            "displayName": "Mark Charsley",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mAuhiLGs9-CmN1BMtie2tWn3oucS__w2c4gNC6yBg=s64",
            "userId": "18064761308611293318"
          },
          "user_tz": -60
        },
        "id": "JSitXn3PyIIe",
        "outputId": "794d10f9-8f20-4982-8326-96bc52a2388f"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQsAAAECCAYAAADpWvKaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsnXt8W/V5/9/fo3N0s3yJ7cQB50Kc\nG4QmhGsIGU25rjCgpYSVMiCllCT2YMtY1mZdflu2ZV3apjQb1E7CbUBL2xGWFlpouRZYlgYICaEh\n5E5CQuzEdmxHlmTp6Hx/f5wjWbJlW7JlW7bP+/XSy9I5RzpfWUcfPd/neb7PI6SU2NjY2PSEMtgD\nsLGxGRrYYmFjY5MWtljY2NikhS0WNjY2aWGLhY2NTVrYYmFjY5MWgy4WQogvCiH2CCH2CyGWD/Z4\nAIQQjwshTggh/piwrVgI8YoQYp/1d5S1XQgh/tMa/04hxAUDPNbxQog3hBAfCSF2CSH+OlfHK4Rw\nCyHeEUJ8YI31n63tk4QQW60x/UII4bS2u6zH+639Zw3UWDuM2yGE2C6E+HWuj1cI8YkQ4kMhxA4h\nxHvWtuxcC1LKQbsBDuAAUAE4gQ+AGYM5JmtcnwcuAP6YsO37wHLr/nLge9b964GXAAFcCmwd4LGe\nAVxg3c8H9gIzcnG81jl91n0N2GqN4b+B26zt64BK634VsM66fxvwi0G6Hh4AngF+bT3O2fECnwCl\nHbZl5VoY8H98hzcxF/hdwuO/B/5+MMeUMJazOojFHuAM6/4ZwB7r/nrga6mOG6Rx/wq4JtfHC3iB\n94E5QD2gdrwmgN8Bc637qnWcGOBxjgNeA64Efm19sXJ5vKnEIivXwmBPQ8qBTxMeH7W25SJlUsrj\n1v1aoMy6nzPvwTJ7z8f8xc7J8Vom/Q7gBPAKpmXZJKXUU4wnPlZrfzNQMlBjtVgLfAswrMcl5PZ4\nJfCyEGKbEGKRtS0r14Ka7ZGOBKSUUgiRU3nyQggf8BywVErZIoSI78ul8Uopo8BsIUQRsAk4e5CH\n1CVCiBuAE1LKbUKILwz2eNLkT6SUx4QQY4BXhBAfJ+7sy7Uw2JbFMWB8wuNx1rZcpE4IcQaA9feE\ntX3Q34MQQsMUip9KKf/H2pyz4wWQUjYBb2Ca8UVCiNgPV+J44mO19hcCDQM4zHnATUKIT4CfY05F\n/iOHx4uU8pj19wSmGF9Clq6FwRaLd4GplnfZiekUen6Qx9QVzwMLrfsLMX0Dse13WZ7lS4HmBJOv\n3xGmCfEYsFtK+WAuj1cIMdqyKBBCeDB9K7sxRWNBF2ONvYcFwOvSmlwPBFLKv5dSjpNSnoV5bb4u\npfyLXB2vECJPCJEfuw9cC/yRbF0LA+l86cIhcz2mB/8A8A+DPR5rTD8DjgMRzHncPZhzz9eAfcCr\nQLF1rAB+bI3/Q+CiAR7rn2DOU3cCO6zb9bk4XmAWsN0a6x+Bf7S2VwDvAPuBZwGXtd1tPd5v7a8Y\nxGviC7RHQ3JyvNa4PrBuu2Lfp2xdC8J6ko2NjU23DPY0xMbGZohgi4WNjU1a2GJhY2OTFrZY2NjY\npEW/iIXIwcVhNjY2fSPrYiGEcGCGY67DXND0NSHEjB6es6i7/bnGUBrvUBor2OPtT/o61v6wLC4B\n9kspD0opw5iZb1/q4TlD5h9uMZTGO5TGCvZ4+5OcE4ucWVhlY2OTPQZtIZllEi0CECgXFiglg54d\nlrj4qjvcIo9CR2n/jTfNcaTzPLcjn0JnmUzaH/9r3jdUhXCBwHlaInSJkBJiyXpCIIVAqgLpEOge\nUIPgCEXBsI5LPB6S76d63A29+d/2e2JhN6/vxkuBKB70azcdUo01RCth2ZbWBdcfYpHW4hQp5QZg\nA0CBUiIv1b7YD0NJH+HUBvX8iQins3dP7PAehNbhPWnmxy219o9del2ExuahexV0t8B3rA1HQI/v\nNzQFqSnoHhUlYhAq0cg/1ErreC8F22vNgyI6MhJJPlc4+bEMh3v3ntJEdjhfv5wj0r/vYTDYKl9L\n+9j+EIv44jBMkbgNuL0fzpMVhoVIQNpCASAielwwRCSKqyGEqwGOXZFPsNTD6A+CprVRqNI0xUHB\nJwb5h1oRkSjOhiAi0EZBfQtEdNJFOJ39Khixz7E/RUNozmEpGOmSdbGQUupCiPswqwY5gMellLuy\nfZ5skEtCMSgkCIbUHJTujGCoAn+5i8ZzBbdcv5mtyy7GEdQRkSgiEs1IIIYjI1kwcmIh2UBPQ3JR\nJPrVqoAkywKsqUh8WuIAIOrViHpUAmUa7sYoWksYJWIWiFIC1hckoiNigpEgHJ2mIdBpKgL9Px0x\nz2FPSdJlq3yNFtmYls/CzuC06YR/nIIabBcCEYmad7JgVfRJFNM+R///GAjNidD6/73kEiNKLIRT\nyzmrQjidg2NVxO874n+DY92ESjSCYwzC+RpSU9qFogdSnrerY/v6ntM6x8B8ziNJMEaEWOSiSGSF\nXghFqu1Sc2BoCkevlZyarvCPN2wkXKDE92VrfB2xBWNoMSLEIlfJ5pcl3V/2RF+FXugm6tXi4dEJ\nv4a8Y5I1G/4cNSQ7PM/RWXg6PM7Euog/xxaMIcOwru6dy9ZEn78kCe+tJ4sicdqBpmJ4zXMHx7pR\nwhI1GEX3OAgXKER8Akeo/fCoVwVUDFUxw6Yd/Raa2mdfxkCEVQfC6TncIyXD1rIYKUKREk1FJtwS\nt0dKvfzo2fUc+rIP3S0IF5j5FOEChV+sXkPLBW20Xt6Kf6wD3WOKRKjEib+8mzEn5nD0wroA28IY\nCgzL0OlIEoqkL2csFOp1J//aJ3yZDa+TSIEzbknUXRvhB5c9G99/i68FgAtXVuJukii6pO7WEBPX\nKajNbfFci04WBsTP2VM2Z0/0l5UxENZF/FxDxMLIJHQ6rKYhuSwS0I9CkZS+7aZuXjGjt/vbcyNi\n+yxnpSOoo+gGoDHq/5zccm1L0nHb2sK0lgv8E8HZrFD4ipemydBwnouzN5yyXiuW0NVBlCI6QtNS\n512kSX9NSwZqOgLDc0oybKYhI10oEqccvs+itI73cuq8URheJ1JzxIXC0No/8lCxg3CRYFV9clOw\nC11OPr63mrKLavEdNfCeiFJ0IMS0n7S2Ozo7nFdqKtLrNq2aHsaeDv01LRnI62S4TUmGhVgMa6Fw\nal1/2TosDJNeF1JzoAajhH0K7saoFelwJNwUoh4V3WNZBjqsKP045csfO1lE82QF/5nJ4dN4ODUm\nGpqK9Lo4dd6o+OPe+i4S6a98DFsweseQn4YMe6FI9ZqaFndiGoXe+NqOuovzKTrYbmb7z1QZ1RKO\nh0YDZS4ADBWcLQaKLtm5rKbL0ytH3eQdk3hPmIlZhqag0L6WBIifd/SOVrx1EaTmIOUE2Kll7LuI\nv99+jpb0N8NlSjJkLYtcT7Tq069iN9ZE7Bdbet00zx5NpMBJaGweh77sw1cb5fBtBoExKrob3E0G\nkQJn3JJQAwaL/2UjLRNNa6F+lqDi2SVMen4Rvw24ks5zxydfwHPC/NrrXlNojv9JHlGvStSrITUH\nUa9G63gvakjSVuwyF5wF2swXyJJ1EX/fWbYyBvraGQ4WxpC0LHJZJCB79Sg6vW7ily+iU7jrFFJz\n4K/I58y3dZSIwaSnIVAmafp8G/7DbsZsg6PXCsa+Zf4uLPB9RuDOX/GD96/lwJVPdHmuPynax0+W\n/Z6Kl++h4H0XjpBE6OAvd+E71kZUUzBUBUMVeE9EcZ6OxBeddfv++uhgzKaVMZAOTxj6FsaQsyyG\npVB055eIvW6HX2kR0a0QZhRPbQhXQyghygHlo5sou6iWQKlC2YRGQkUK5ZX78SpOlhQd61YoAJYU\nmfWKDl77GC2TDbatrEHzS5wthiUSCoZmFszRvQqv/OwJM4GrQ25HNq2L+Gtm0cqwLYz0GVKWxbAV\nikxJTLSK6DgC5q+jAhiAGpK0/OYMWi5og/MMDs7eCLNh+hOVMPnVjE938NZ1AISLBGG/QrhAwdli\nLV3XJbpbcO5DVXgnS0qb20yfRUJINSmUmgXrIv66Q9SXMVQtjCGTlJXLQpGxSGT4XrqretWxLoXU\nHJyelMfmteu4ZveNvHLOC5mNLU1mr64ikge77q9m+hOVTHg5FC+956r1IwKhXtW76C19FY2BnI5A\n7iRtDbt6FsNGKNKYbnR6/V6Y8e6GCDPXVvWLUFy6YwEABX92nMBEndmrqxj3ehgRMczyfLX+lGtF\n+mM6kvT6A1AnI5sMxelIzotFrgpFRvPmXogE9O4LFku68tZKZq+u4tIdC6iPtmb8Ol3xh9kbAXhr\n5iZQDcZ95RDhQtOiyLjsXi//L13RF8EYjOtsqAlGzopFLodG07ooY1+EXr6HLoWii/oUUnNQe1kR\np8/yoHscOP0GO5ZXM6WonlJHXq/G0B0BI4znsJPTa8bjqQ211+jsZqxdvqcsC0ZvRcMWjO7JSQfn\nkBeJvpyjD+Z68Z42/OVOAmMcqCHJ+auq2L6iuk/jScU5m+/E9XY+o2rNxC5oz+oUXS0ys+hy3Ujs\n/zYCnZ9DxeGZc5aFLRTd0I1VITUHLROdBEsFulsQKlL6RSgAds97mh3Lzdc2VEHrOA9Rr8rhG4vS\nen637zOLU5PeWBm5ev3lAjklFrn4QaV1wWXhAu+NRZGYz1A7N593V9WQf9RAfLEB/4T+jXLNXFtl\nilKxg7BP8OmVHsa9lsI30oXA9fh+sywauc5QmI7kROi00FEqL/XeMNjDSGIgLAnIQCS6K7prLSAz\nq1pB7RwPHy7tH6sikalPVaJEwXcYiveEzHoXVrp3p6lIiqlJr5ax92Gaksm0ZKBDqfHzDvB0ZNiF\nTgeSgbQkeisUqfaJSBRHQEdEDIo/0pmzvJJpby7s0xh7YstfrMHVIPB9Fo0vUutpnIn0yj+T6DjO\nNAydgYUxWFZuLlsYOengHCz625ro1ZcjxZdMdicewOlFzUT+twRtZx7Mz/yU6TCjpor8w5KSExG0\nljCeOit02l1NzhT7+looJ+Xn0Y1VMJQcn7nGiBeLjHIlEp/Xz0lGQNdFd1PsB6wvq0Lrh8UoF5/m\nwvJP+21oQoeTcwzyDmuUvxHqFDZNWUkrccwd0sGhl9OSVPQgIOkKxkAvNIuf17Iuci1CMqKnIb0R\nih6nDwkFYTrd0iHF8ekIhXmcGb4sOAClP/Oyc+MM7vjkC+mdN0N23V+N74DZH7Xj+Xsce4djYmRb\ngJM+q45in+Znn4tO98FiRFoWGadox56XZqJUtLSAPYtdnPP9U51rVGZAJl+0RIoOmOs0/BMF+5tK\nMzpnOpz7UBUFhw1KGiNJbQ67YiCtjMTPKDx5DNrxFkTEnP7EXfmWtZDrU5Jcy78YUZZFxnH3jkJh\n1Zk0Sgu7tACkpmJoChU/kxiFXozCzLInU5bwj5HCQkmsWJWIEjHQ/II/zN7Ig40VGY2hJ3bdX82W\nNeusFO/kGhaprIv2fcktCqTX3b4Qro8NizpZfJqKdrwFvdRHeEJxcjGexM91CIRVc4Vhb1n0ddl4\nTCQiZ5gJR5ECp7myMmGOLjVHfM4uNQeRAicNMzRGf2BW0haR9tL8HX9dO/b1CI4vQIkYOBuCgNnZ\nPL7uIgWJIpFYjBdg7JYoFx+o5N1VNdRHW7OW9n35fYtRAwaqriM1BYP2X53EkntJ8bjY+05YIeuv\nyDfrbWxuNGtzdDi219OShP+pIxChcVYhxc15RL0a2pF602pJWCrfk4UxWL4LyC3rYthaFr1eI9DF\nHFWt9yM1hZYJ7Rd77G/9BYXohW4Mr5OoV0P3mv9WRTfitSZidPx1Td7nQIkYnLjQRWhsHnqhi+Yp\nXj69Or+T5RAra2doSvxmbm8vTAPg9EvmLK9k4+lpPOcvyPz/0YE5yytRwqZBb6jtl0/7+c1s0qPX\nFsUriwMdKoKbxyhhie+zqNkhLfH/kYmPJw0K9wcIjc1DbQ4l7xgi/ohcCacOO7Hoc+3LxNdK+GUT\nER3do1K8J8ShO8FfkR/vFer7TCfqUYl6VU7OzuPth9dT+En3i6o6Ye13BHWK9kc5PlfjwN0K4raT\nFO9JrtId9WoEVvn55H54eeOTSKsgb9TqINbRwgB4ZO1N/MPP7ujd/8Xi4hWVKLrEcAoMVZgipSrx\nOp9RrzkFi3pVCj4xaB3vjdfrTLzphS6CZR4Mp+DIDbDnHg9tY31WclmypZWxaHR8PhC16mzohWab\nglQWS0/XjO3oHCbTkKzMO9O4GDyfms14pmxwc/osjZc3Psm8pUsAMFQH/nEK/gqd+YsW4QyaCVKJ\nFkFHMxtIurijXo2oRyXsU9DzDPIKg5w4UYi8PspZG51xZ2KgzEXDH/LYe6+ZpRkqcRIqUmi8MsSB\nK59g7rIl1M2THLx5PQDnr6oi6gat1ayWFS6Jcu0FH7J+3JYe33PFpsVoTQrOZoHDZ1YGr/zHjTz0\n/Vt5d1UNc5ctoenmVnbPe5p5S5fgbIny7CNrOaxrXOgyP5drvnY3joA5dkNTCOdrvLlhAzO33o7j\nQAFnzqzD0Erj/w8JXTuGu8vhsIgJjuF1EizzoAajaC1hlOZA5+fZ05G0GdJikTXnVE9CEdHbE4o0\nFUcggu+YwvxFiwhMUGiebpB3TOGqr77D2jPe44pf3gvQaT4vNYfZJawLJ+Whm7xorYJ/vfMnfOfZ\nv+DDOc8AZitB3WsQ+7g2rzXL3K1rKmdJ0THCPoV3V9XEMzYbzxVU/+kTXLpjAX+YvZHSWz7tdSGc\nmOA82FjB/52qYKNVlu9fp5n76+bAwXlPA3B8vuSMNx1c90/LeHdVe4uB/Xc6mPI0iIhh/srrkrnL\nlsA4hdIvnOCcUbV85B3Twfejpl692oOlkTiV2ft1L0pQkH9Q5ZKvf8Snd5zZY9e0XI6QDLZgDMm1\nIVn1YPdQcj9ObE6tqXFTWveohAscfG7ZB/z+ldn4Dps+AmeLgRqMxovnxsrNHbzVwfTHgimdlVJz\n0Drei+42Tfy2IhFf2Tn1qUr23VXDvKVL4jUvt6xZF3/ujJoqPqrs/7UgqVh8dG7cQjln852438hP\nWu164cpKvPUGzmadcKHK2h8+xIUuJ+evqqL0lk8pdAbZu2ka4UIIT2xj2o/DpqAmhlO7WfLeVdRI\nag7CJR4OX6dRuFeY7RybA/FCx9AhPJtgMXQnFoNlWcTPn2WxyOraECHE40KIE0KIPyZsKxZCvCKE\n2Gf9HWVtF0KI/xRC7BdC7BRCXND7t5FiLNnuUNWLeWhs3h3O1wiMUXnlwf9ky8/PR1YEaJ4mCfsE\no799kE+/afo4dI9K9G/qOXFfEK2wjahXTemsBHA3hOOFcCMJgYt9d5m/0mGfufS88VxBxcv3xPcP\nllAASVOZ3fOe7rQsPuo2l8zrHgdhn8Ktz/8VANtXVPPKOS+wcfKr+CcafHxvNb6dqdeXdAq5JoZd\nO5IYCQnqTHwpQrBMcOAWX6djuoq2dHeNDbbvYjCdnT1aFkKIzwN+4Ckp5eesbd8HGqWUq4UQy4FR\nUspvCyGuB+4HrgfmAP8hpZzT0yC6syz6LQ6eYel9NBWjMA9/RT4Abz+8nrnLlqCGJIYqaJqsYFxw\nmlCziwunf8IHm6dScNC0NEJFCqcrDIzSCFIXjHvRgafW9MwrESPeMcxQFXSPg8Z7/PEpSCJnP1JF\nW3mYQ9c/mpV/wUDynL+AVWvv6LLGxvxFi3A1tsXDxF2Fig2vk7qL8yndFcQR0IkUOONh5sTwNZg+\noNNneQDI/ySIIxBpL/2nqRheJ47jjaaF0cFiyGXrArJnYWS1i7qU8i0hxFkdNn8J+IJ1/0ng98C3\nre1PSVOB/iCEKBJCnCGlPJ7e0E36PVEmU6GwOHRzESW7zF/+i1dU4mu0nHaqwN0g2GbN3QGOTPwl\n19V8i20ra/htwEXlqwtx54UpfjYPNRBFWhGLqKbE+44aqsBwCpwvFXKOfie7E14P4ON7B8+C6Cu3\n+Fr4h7LU+x5srIhX3ILkvJVEYiIQLINPJrgp2ms6XEOXuCh/sxUlkpzvIjWFunkSx2kF3zElntvh\nO3gavdCFVp/C4WnRne9iMB2dg0lvQ6dlCQJQC8Qug3IgcfXSUWtbWvRXI9wk+tCnY9KmJvKOBvHW\ntTHq4wBqUEcN6ii6ZNvKGmZuvT3+lAmqjzOvNv8VDx+9kkM3bWD3vKdpmqww99+3xnMUDNV8/Mbj\nj/DgQw8T9ikoOmhb8vv+XnOMrsTuqQ1mGwhphV1/+6unkzq/m/va75e/1cbYdwwMFYJlIv5cM2yr\nteefqArjX5K4GwW6RyVS4OTth9fz0ovPYKhKp+ngUGIwpiN9joZIKaUQImMvqRBiEbAIwK34Bibt\ntq81KAJtqFaGotQcGBC3EL5TN4vWIwXm5MsiFoH49bSX4ttifTbGFBooYTNnYa9/DA9qFTxQfJDG\nGe0+iuHOtDcXou3Mw6lDuMBBqNhB3TxJxbNLEMvamPofeqcM1ZgoNJzr4KPKamavriKv1iCcr6Ho\nkiN36OS/4yH/aDT+/3U2SYJ/dYq8HxZy7YKFOAIRXIGmbh2nYFsXHemtWNTFphdCiDOAE9b2Y8D4\nhOPGWds6IaXcAGwAKFRH939IJk2h6DHFODEt2fr1AnjjB5dR/S9PsODA1fHwYpcvUR7G+EDDUAXH\nr4/g/9k0Nq4wnzNShAJg7/wn4/U2Zq6tAsBxWvLQgsf5bfNMPtLOjR8rNYXaOR4MB/iOybhTt3mG\njtBVWi5rY8yLLoreUvFPlDhCCsEyQajUIO+YIM9hTk/afSIZtCzIUQY6lNrbacjzQKwM00LgVwnb\n77KiIpcCzZn6K/qFvlgU3cT1IwVOGqe7mPvvW9Hdgi9623oUCjD7h+puQdgnOHjtYzz6d2t7P75h\nwodLq/nNfd9n3101fNHbxpb/uJioR41npxqqgrtesuv+ak5PFEx6fhEAh27awPYV1UhdoXWswu/+\n3xr23F1D+LpmPqqs5gfXPYO7XiIeGk3DuS4iBVatiO6K9CSQy5GRgaZHy0II8TNMZ2apEOIo8E/A\nauC/hRD3AIeBP7cOfxEzErIfCAB398OYcwatJYy7ycHPtl5KfpF5AR+6aUNazw37BKErTgPEMx1H\nOhPU9vCmoks+udvgrCcs346moIZMAzRVqPjgtY/BtQBmzPnDOc9QsWkx5a9BQXMYRTcoPAiHbtKY\n8gs3kQIn3j9+1u/vqb8ZSOsiN5Ky1NFybuHN2X/hLPYUheTsQMPr5MAtPvI+EzTP0NMWCYCKZ5cw\n+XPH+q0P6XDiwpWV+D6LEi5QkhLR0mFGTRVCh7HvtKG1mF8oQ1M4fZaH4i3HzRoXXSRmJZLrYVTo\nfSjVLtgLfV9R2E01bTBj+hNeCZN/NMq3P/+bjF76na/80BaKNIn4BOEChbBPJEWbuuOpllKmPLOE\n0NgoBYcNtJYwIhJFCYRR6/2Meu9kRmOwpyImw1MsBqivqNYSRg0YPL3yRuYsr2TqU5VpPa8/2gkO\nV3Yuq46nwKdKVEvFXQX1FO4VTHzBIP+TYPuOxJoiw8DBmchAhFKHn1j0VShSVqPqvOw5/rAljLeu\nDaffYN9dNWkLhk36vLuqhmAXCV1dEcu/iBQ448v7IXmdSbYKBI8U62J4iUU/fGiphCJVMo8aMLj8\nvsWU7pSc+1BV1scx0vnpXelHjBYfnctHldWEih3UzzJTvpv+ubOFYZMZw0csstGtvBuHZndCEUMJ\nSxQd3A2SmWurqNi0uFdjsulMJhGj2OK2xnMFQgdvXYRR33H2nFvRzTU0FGp19vdUZEjXs4jTX0Jh\nLXXuuEApdj9xARiA7lU4Pg8O3jpyEqtyGe8xgbvJIFCm4ar1mxv7yaoYCRmdQ1ssstUdLG41qEmP\nT84pxlDBPxEm/iaAklDJ+tQ5Zk7A1tU18dL42sI6Ds7c1Osx2fSdKc8s4evX/p7XvnU5pUFzZWrH\n+hjDmf7Muxi605B+FgrD68RQ4UfLaxi7JRpfkxC7uRujBG5uBiAyq5XWsQp1743t9Zhs+s7k1+8m\n/6DCx/6x5iK/5rbMhWKEOCt7w9CzLLLQkLgjZiEVB42zChm1209biZtwgQM1BJ83a7xiqEonZTXe\nKeKGUdclrXGwGVhmrq1C9Zt1Q8boEm9dGyeWTcSRTip3hn1Wc7nkXiL9ZV0MHcuij53LU7YdjFde\nMoXi5ByzInX5P+7j+HzJVX+zmYtXVKJ7FUIlGp9c5yZU4iScr2E4Bb5jMmlFqc3Ac9mC7WxfUY2i\nS9yN7aUMuyugk0g2WyYO9xBq7otFH0UCurggEhyaIhJl1G4/k/5H57OvhAlFVSaf/Rm/+u8/MTMI\nfYLT4xQqb/gdx66CULED3S2Yfd8Opj5VyYIDV/dpfDa9Z/24LUx6fhFb1qxD95qVw6G9/GEnstiP\nZKSRu/+5LKh0ur1JY+geB+eM+4wPNk9FL43w4uI1fGlLJdGowtb5D1PqyONhXXDyyjCF77jMEN1d\nPZfTt+lfYutyAqUKEZ+gbFsEEtowCOjWb9FpOpLQHiBTciUq0h9TkdyzLLJkSWRqXhqawulFzTRs\nmIiYGMB7wMlhfRR75z/JgSufiKdoH7x1Hfdd9AbbV1T3W4dym96xbWUNO5dVY6hKvOHRnntTVByz\nrYtekRtiIUTWfBI9ikSqCyWi4whEGPOvThrPFfjyQviOSb7obeM7dbM6Hf5A8UEAfnLW73s9Xpv+\n4cKVlYRKNLMR0+e8lG0WndsjQqe0/k7XzTDwP2Q7SSs3xKKXpC0QMbr6RdHUuENM9QsKqwtw+g1m\nrq3iu2U7szdgm36neZqk9qY2Nq9dx6Q793Hi+jYaZvq6fkKCaGTL2TlcHZ1DRiwShaE304xUQpGq\n70T5m61mjD5g8OHSoVtNe6Qy6fxjHLjyCQA2Tn4Vp0unrUiY/osuGlIDcdHIZnRkuJEbxW+cZfKy\nsV/rvxN0YVF0TMSKF+K1mgw3TncRdZvLpG2GHrPWmMV8C/e1JodSUzg7pddFYJyPvL0NiECo3eGZ\nQT+RTq+ZA45O6L4wjl38JpE0u5cnCkWMgiM6ebUG56+qYkaNvZJ0KDGjporCT6J46yJJHeiBzteE\n9dh/psN2fnZDboiFEP3zIXV1UG2xAAAgAElEQVTzmh1N0cSFYbHy/obTbBe4fUU1l9+4Pfvjs+k3\nPqqspu7WEOFCK31fUwiNzSNS6o1PSWI3qTmIejUKjuicnl4EZDdZa7DJlqMzN8QiRjYFI53XSrAq\nDj6g0PA5Lwf+3InuMcNubz+8nu0rqpn6VGVST0+bocHe+U8SKlLiVcIB6i70EBjnw/A649ZG1Ksx\n46FdVme53E/nHixySywgSfH79PxuSFx+bnid+CvyGb3Jw3eW/ZSyzYJwgYNwgYOAYV44I6mXx3Bi\nVf3ZqCFJqETjlZ89QcsEFf/kKJ9eJ2gd7yXq1dALXUQ9Ki//8hI8dUG04y3J188wjWz0htwQCyF6\n7IjdI2kKTOJ5wiUe2krcNE1xcGq6wrLXb6N1rMLmtetw3FPHxT9eaqdyD2FWlH5M2CfYvNasCn7R\nN3dw8Ob1eI5ajZM9arwZtaPNelIWrdvhFkLNDbGwSBnW6snSyMAKSYx+xJxdbzz+CFd99R0+qqxG\nRAX+WeZV89bMTQQm6mk1DbLJXWK1OKG9gtau+6s5Pc7B4es0woUqoWIHxR/ptJW4zemJ1z1Yw+03\nsuG3yI3QqecMOXfKPV3m7/e11VwqAYrNVZuneGmaDpEig9su22InYQ0zvlM3q9vP9MFGs8fsnOWV\n+I6F0VrCOOpbknuKJIRAM12inivhU0gdQs0kdJobYuE9U1569r1mHLynZrVpCkfKaQ10CpUGyzyE\nih0ESwXhwq47fdsMPyY9v4jSdxwoOqZvo0gw5p1mlOZWZCChwG8fxMJ8Tm4IRl/FIieCylIR6IUu\nqwRaiv2ag2NXFjHudw30SdpSCIjzdAQ1GMXpV3n74fUc0f1JbfRshifnPlSFF3A3md3WnacjuBuU\ntH6whip9XYmaG2LhEJw8z8OY91oRHWoQmNMFFVeTJRNpNrTtRA9+DSUsmbtsCf5xCv5pYQ5d/2jm\n57AZMviOSbwndLSWMFJTCJS5KNxxskfLdahUy+oPcsLBqXvMku2nzvbGpwex28nzffjLXTj90lw9\nCJmHVlMkYMWSsI7ep9NwriveIu83933fFophSGIG7sUrKjlxWRQlYuAIRNDqAxS9e3xYdirLJjkh\nFmNKm9i+opqyuz5JKoob9ZpfcjUkcTeY875u03ZT0cUxsWxN9d18InmwZc06Ij5hT0GGKY4Lmrj8\nvsVc8Y17KTgcZupTbTgbgsnTDutvtjqVxcilEGpfoiI5IRajHREqnl3CycfPoq3YFU+5NlSFYJmg\naYqDUImWtG4jSTC6WUXYkZjYHP+TPPzlLj5cWk1wXJTn/AX2grFhjK47aP16k2lNBM36JcPZP9Ef\n5IRYgFmBautqM1NS96jWzcGoPQbuesnmteto+JzXzLjzavHsy1jablI+RjciAaZV8eHSak5c38bM\nrbdz8Ob13OJrGdD3azOw7J73NK6fjyJcqPLyxieT9vU1ND9SyBmxAFjXVE64QOGNxx9B9yoYToGh\nQsQnmP5EJdtW1uAvNy2PqFclWOZh/1fzOvk5YqLQ8XFsG8A1X7ubcT/XYEvRoLxXm4HHd88xxOIT\nXPGNe5Mqf3cZZrdJIqfEol7PJ1SkMKOmitavNxH2KbzwvR8SvaKJsotqmbn1dnS3iFse/jMd7L99\nXVKX7BipKjt33KboEm+tZPZquy/pSOCVc17grZmbMFTRbc9am9TklFg8/v48tq+opuKqQ7x/0S8I\njBVc+ou/5cM5z/DWzE20fpaP/zq/2cej2ME5X9/N/EWL4isKu7sAOu4zVIXfPPpj2ooEO5ZXc/Dm\n9f363mwGnwtXVjJ/0SKcpxMcmPFFhX1YvDhCyCmxOHjtYwDxxj0fLq3mhivea99/83rUd03rw1AF\nH/xyBm9u2IC/3BWPoPQ0DTE0hbZiF02TnbwUKGXHctupOVLYtrIGwykI52tEvVrc32V4ndR9YUzq\nJ2Up+3I4RER6FAshxHghxBtCiI+EELuEEH9tbS8WQrwihNhn/R1lbRdCiP8UQuwXQuwUQlzQq5FZ\nrD2jXSyu2X0j02/cy7urajg5xyBULHkrBH+2/PdECpzxW6ppCRCPpqjBKP6J2E7NEcasNVW8/fB6\nWiaYTs6oVyXq1eIhett30T3pWBY68LdSyhnApcBfCiFmAMuB16SUU4HXrMcA1wFTrdsiIGvFIF45\n5wU2Tn6VxUfnMuPcI+y7q4bPu2Hjo1fy7Yef4tQ0cxVhxybGsRsQn7KM2WZw8YpKPv/hzdkank2O\nEwuNb19Rze5wAN2j0jrOwys/ewLFDoj0SI9iIaU8LqV837p/GtgNlANfAmIxqCeBL1v3vwQ8JU3+\nABQJIc7I5qDXj9uS1GO0eYbO3224h+ZZYS7/f1sw1PbSeEnvJVYuT1MI+xTqL4nib8tubwWbocE5\nTi9NFRonLhDMX7SI0vebO4dQc2QBWK6Qkd0lhDgLOB/YCpRJKY9bu2qBMut+OfBpwtOOWtuO008c\numkDd3zyBRaN/T1ff+sexnscqMHUSmioZmjWxmb7imqmvbmQxsY8dG8BRe+2DvaQBozeLCpLWyyE\nED7gOWCplLJFiPZVrVJKKYTIaEGoEGIR5jSFCeV9nyvGuoP5dro4eluQsc+74iniim5gqO3Scfl9\nizk9zmE7N0cwc5ctwVAFeT5By2VBxv+yxc7m7IG0vqVCCA1TKH4qpfwfa3OdEOIMKeVxa5pxwtp+\nDBif8PRx1rYkpJQbgA0AF53nzkpRje/UzaKtRKIedgMSQ1NQIslCEfNd7FhezZRnlrD/9nXZOLXN\nEGD2arOPSMO5CmUtOv4zHZS9e5oz3gwPSBZnrjRN7i3pREME8BiwW0r5YMKu54GF1v2FwK8Stt9l\nRUUuBZoTpiv9ynfLdrLn7hr23F1D61iFYwvDhAvVZEenKqi7NQRgC8UII1wIp6YrTPhdK96jfso2\nN6I0BxCBUOdFZEP4S91fpBMNmQfcCVwphNhh3a4HVgPXCCH2AVdbjwFeBA4C+4FHgEHpzuOfYLB3\n/pMYqki+OQUlL3gGY0g2g8xHldWM3q6jRAxzgz3tyIgepyFSyv8Fuiq7dVWK4yXwl30cV5+JWQ1h\nn0DRk4e/ZY1tUYxE5i1dgjtodihzRKJmb9NAaLCHNWgIzQkZ+DhzKoOzP3h3VQ26u92y0N2CC1dW\ncsF7Xx3sodkMMJvXruO1px8j6lEJjc3j0z8rMXf0Ux2L4cawFwswLQlTMMzH21bW8P5FvxjcQdkM\nCmc/UkWgTMN/psrYrcHUB9n+ipSMCLEAMFTilsXZj9hNjkcqH99bTf0sQdGBkNl9zMK2KnpmRIjF\nDXuvI+IT6G7i1oXNyGL26irWNZVz8YpK9t1VgyOgD0rNzVxaUJYpOdE35KLz3PKd343v+cA+cET3\nc9OabwHYyVgjjOf8Bdzia+HaBQvj5fTijs1U/ooepiF9re6dS7kWfwi/NLT6hgwEE1QfkTxQorA7\nHOAcp3ewh2QzQPzjE3fw/WOSAi1ZIGzHZmaMiGlIjHChJOqC0Y7Bt6ZsBo5d91dTsPAooRKnXSGr\nD4wosdh3Vw3hQoNSR95gD8VmgJi9uopJL36TW854H8DM2EzwVYx4q0KkNQMBRtA0JEa0yM7aGyl8\n/sObyas1GP1BlP95/Grym5t6XgOSQ/6EXGNEWRYAvpIUzVRthiVvzdwEQKDMhdocsn0VfWTEWRYf\nznlmsIdgM4BsXruOt0Kw+ou32O0J+8iIEwubkcG0NxcycZ2C2tyGCLTZjYSywIibhtgMf2aurWLa\nGSdMoUhoJpRoVdhTkMyxxcJm2OGul+h/VxrvZWpbFdnBnobYDBumPLOESS+0UdrcbAtFP2CLhc2w\nYNaaKsZ/FGmfetgikXVssbAZ8iw4cDVl24I4ArZA9Ce2WNgMaa74xr2oQR1hlcpLcmjaZBXbwWlj\nY5MWtljYDGneePwRCld9Gu82Zy8U6z9ssbAZ8myc/Cp1F3poK3EP9lCGNbZY2AwLdi6r5o3HH8Ff\nkY9e6AbN7oyebWyxsBlWvP3wek6d7cXwOuOCYZMdbLGwGXaESgXqD+pN/4Wm2tZFlrDFwmbY8eHS\navYeH4Ne6Ep2eCaIhtCGbuHcwcKWXJthyd75T8J8837SEnWbXjPixGLm1tvtmhYjiHlLlwBQqJ1C\nYvXhtEWjV4y4aYi/wa7qPVL4/Ic3A+Cta0uKkMT+2lORzBhxloWzzr5ARgpvzdwEa837k55fxFm/\ndOL5lO6bITs1uw5nF4woy2LqU5U4QoL6aOtgD8VmgJky9Ti6V8HwOs3oiG1dmGTQZGxEWRbOZoES\nhZNRQamdFTxiOPehKnzHJAUNIXuhWR8YMZbFEd2P1gqOEHY3shHGv9z9E7auromvTLV9F71jRIjF\nDXuv46Y138IRkjhC0u6iPsK4xdfC7NVVfOXRV6m/oJAXX/kF0dKCwR7WkGNEiMWvp72E5peoIVDs\nqNmIZMfyapYUHePdVTVMfaqSqFdN8l0MFLnUFDlTehQLIYRbCPGOEOIDIcQuIcQ/W9snCSG2CiH2\nCyF+IYRwWttd1uP91v6z+vctpIeig6JL1JDk43vtLuojlbMfqaJ0p6RpspvIGe3WhT0V6Zl0LIs2\n4Eop5XnAbOCLQohLge8BP5JSTgFOAfdYx98DnLK2/8g6btCYufV25i1dghqSlmDAhSsrmfTiNwdz\nWDaDhLMZfMfaKDoQavdhdDrIFo5U9GiDSSkl4LceatZNAlcCt1vbnwRWAjXAl6z7ABuBh4UQwnqd\nAcf9QiGKnnxRbFu5bjCGYjPIzFu6hNENbYiIgSMQaY+MaKpZCVzT7H4i3ZCWz0II4RBC7ABOAK8A\nB4AmKWXMA3AUKLfulwOfAlj7m4GSbA46HaY8Y6b5Ov0SJdx+A5i7bMlAD8cmB9i8dh26x4ESsygi\n+ohekSoj4YyOT0sspJRRKeVsYBxwCXB25kNLRgixSAjxnhDivZMN2Y99+44oTHtzIYouk29hScON\nwayfzyb3mVFTxcnzVQyrBJ9d7yIzMoqGSCmbgDeAuUCRECL23x4HHLPuHwPGA1j7C4GGFK+1QUp5\nkZTyotEl2cmQ+k7dLKY/Ucn0JyrJqzUof9KJs1lHiRjtN11S9qxZfi1mfdiMDJzNMGqPwZE/zSMw\nzkfdvGKMQi/S6+6cc2H7LTqRTjRktBCiyLrvAa4BdmOKxgLrsIXAr6z7z1uPsfa/PlD+iu+W7cTV\nINAnmrn/MXMz0WcR2zZ7dRX7b7d9FyOJHcur2bx2HR/fW024QEENQd3F+ez5e++ATEeGctgU0kv3\nPgN4UgjhwBSX/5ZS/loI8RHwcyHEKmA78Jh1/GPA00KI/UAjcFs/jLsTd3zyBRaN/T3+WW2M/7mG\nGowki4R131AV3n54/UAMySaH2bLG/KGY9uZCvNvzaJ7toujd4/by9W5IJxqyEzg/xfaDmP6LjttD\nwK1ZGV2azF5dRdQNF/3Vy4zf6EANRnEEO3/oUlNQdIMrvnEvLRNUWipg3101AzlUmxzh7EeqGLXH\nIN8naLxAZ/xvmjsfNIxXoGbq3IQhmsG5+Ohcbth7XdK2Hyx6jMu+vxTDKTqFSmPE4upKxMBbb1C0\nt73mgc3IYtPCNbgbo7ibJIdu2sDJOcUjOjKSDkNKLK7ZfSMLDlzN+nFb+GjXBKY+VclbIVjwzdf5\n3n13MWpvBGez2cpOSXGDdsE4caHCu6tqzJoHNiOCWWvMNUHnr6riHKcXNaiTdzTINV+7G8PWiR7J\nebFYevyi+P1XznmBPS9M4+IVlYzequBuFHzeDb9Z/QW0lnD8pkSMlEuRY4Khexz4DsNzfnsx0Uhi\n57JqLr9vMQVHdK5dsBBHQMcRiMQbKts1Orsnp8Si4mUzYzw2xZi5topfv9EuFhWbFqNffBp3kxkC\nPe/LHzF/0SJ8x9ri1oOIRONCEbufKBxKxMDV2EbRgTDXeeuZvdpegTpSuHBlJUpY4jwdwRGIoATC\niEgUJRCm7PcnUj8pSyHUXIqE9MZfASAGKQs7iYvOc8t3fjeeVfVn89y6K2krAffFDWgbi/nNv63h\nivfuZZQ3yKmAB/cLhfg+M38BWiaobFtZw7ULFnZpTSQSKwtvaApSU9A9KoExKm1FgpbpUQ7ebEdJ\nRgLzFy3Ce9RvXi+RBKvCut8p5bvDF12GM/+y5apYbJWv0SIbRTrPy6mZWqlqWg3bV6zj8vsWAwY3\nfvtvcZQq1JUVsufuGuY+uwTVinT4PhNMeWYJUyKtnYRCRKKdmuR23GaogsBYwYdL7VWoI4Frdt9I\nm67i0KVdMasX5IxYTHrxm/hKAhS3GFx15z040TE0BUN14GqS+CdIzl9VRemxII6AjohEUZvbmPoT\nuvzgO26XmsPcpimE8zXe3LBhIN6aTY4QWF+OsyWKGtTbrwVsX0W65IzPQrSqeDcVAmYClaKbPohT\n0xUCYwVl78DoHa2ozW2dfBKAaUIm3lKdwzpWiRg0zlDjC8oqNi22nZ3DnHM230nwjlPWdDU5tG6H\nTNMjJ8TiZFTj4K3rGP2NT3A1mkuIRcQUDE+dpGh/FHdDpH21IHQWiY50sT0mMGf8byu+Y23MXFuF\n56iDW3wt8dCazfBDVaPk/VcRhqYQ9ahEvVq8F6pNeuSEWJyoL+L8VVXUPXVWUl5ELKSluwWhEtMr\n3cma6Ikujon9wugXn0ZrNZeta37JEd2f8niboU30/SLefng9bzz+CC0Tney7y0W4xJMsGP1UwDdX\nnZuZkhPRkPzCcXLS3X/LmPdacQQ6eJ41B1GvStNkN6O3NpobezPH7PALIjVH/LUNVUH3OAgXKPjH\nKfinhTl0/aO9fTs2Q4A5yyvxntDRWsJITSFQ5qJwx8l4VCQpItLHaEgui8WQi4aIqGT0B8GU4c+Y\nJdFW5DE39NYZFdG7NTkNp2DLmnUc0f1MUH29O4fNkMFfLvCXaxTvUVDCEm9dG23ji3DV+hHNXVuX\nvQmbDhdyYhoiDIna3IYSCHd2VEZ0RKCNcb+uM++neUtJiu3hfA1/uZPmsxyc/UiVLRQjhF33VxOY\nqBMqUggVOwiUuWia7BzWfoy+TEEgRywLpEQE2jpt7ktIK/G5Sd7uBAvD0BSO3ADjJ9WydeYmAkYY\ncPb6nDa5xzmb72T3vKe73N98TYC8t/MIlgoUHZpmFDDqvbas1ePMpSlIX8kJy4IObpMurYMUVkc6\ndPV6UY/KoZs2cEHppwB87lf3x1POwWymazO0ETvzU26fvboKh1+h7Fk3Tr9k1F6dUXsjFO5rHbaW\nRV/JDbGwSPml7kkYMhSN2HNivpErvnEvr/3iEmbUVCEdEt9OF2AuXfceVllw4OpevReb3MBT1/5L\ntPjoXMDsfZp/NMrEl8xVyu7GKI0zVFwNIXO9SHdd1ocofZ2CQK6IhZRdWxLpkqZoJJ7H2RDE1RCi\naH+UUXsM1lz5c/JqDeYtXUL0sTLe/cu1bJz8avpjsMkpVtWfjdMvmbfUTL5779HZVGxaTHCc+UPh\nCOoouoEajBJ1WU/KYjbncJqCQK6IRSIZTjG6fH43xMJjsRWHvoOnOXlzkO+u+Qvq5kmcLVGcLVG8\nium/mPpUZe/GYjOorCj9GN0tcDdEuOZrd1NwRMd3wMH4lyR5nwZwBCKozW04gjrXfvkdgmUes0tZ\n4vUzzL7wfSEn8iwKXWPlZePuyH79w27mnrJDIo7hdcZXo4JZqzNcqOIf62D7imoWH53L+nFbsjs+\nm35l2psLKXvWjafWnFZECpyoQT1pyQDEcnk0IgVOdK9CwTtHze2RSJ9yLHLJsuhqGjLk8iyQsn8K\npXaTWyE6NJgRkSgKEE8oV834u7vJ4PxVVbSVAJW2WAwVZtRUUbYrirPZvK6UiIG7NmF1csL1JgBH\nABo+56Vss5n4N5w6k2XDXwG5Ihb9SQ/JWLH98UVmEG9CY69KHbp8VFnNrDVVtF4QZnJ1wjKBrnxj\nmoPiPSFEIGSHTLsgN3wWUiIjkW5vfSIDqyU2FYl3rbIZUlyz+8b4/Z3LqnG6dGovzet0XMckPqW5\nFeeRxmHnr8iWVQG5IhZp0GcBSbkCtfO2Y/Pz0D0quldh5lp7FepQ49D2cia/fjcACw5cTbhNxdUk\n41ZFTzk8w2n6kW2G9DQk8YNNa6VgV1OSiI70upCaA90naa5qQdtYzIdLq/lO3Sy+W7Yzi6O26U8K\n9wrc77iY9/wSQkWCMX5J4b5uVhInZvpmSSiG4xQEcsWykNI0+frwT07b4kj1q6KZ9Q1O/L8wxbsk\n/lY3/nLBbwOulELxYGMFYHZBs8kttq2swd0QwVvXRskfA9TNk/E1R0kWRYcQe091N4ci2ZyCQC5a\nFrEPqQ9VlWMffCZ1CZSIQf6GQvKXHab+D2cRmBxmonqKaW9WEo0qbJ3/MKWOPCqeXQKFEZ5+54ts\nX2HX7swlLlxZScQnKNOD8Voo0x853flAu4xer8gNyyIVfbQ0gK4tjS4uFjUYZffRsZw3bx9TJtWx\nYP0yvFvyKH7dzZU/+jvTolAlo1934m4yWHx0LlOfqrRTwgeZ2Boeb73BqL16vGxetxGQBLJpVeTK\nFCTbVgXkomXRkcR/fi+tjZSWRoL/Iji+gONzNUZ/EGXCTwyOF0yhabKCckkT4pUi3E0G3hOS9b/6\nUya+raNEdHSPg1bdZfdKzQFK33Ew7/UlOAPReOX3jv1i4nQQDtuhmT65a1mkIguWRhLWPNbzaQuT\nNjUR9im4GkI4W6IUHTDwbipk+4pqlLBEDUaZ8HII1VpPALD9hRmdeq7aDCwz11ahu4VZerHYgaGa\nl3SsEtpAkitWRX+R+5ZFR/ro05CRSCdfhgiEQFMp+cMJ0FQ8gTAur5OT5/t4y1qAqOidq0IrlzTx\n62kvMe3NhbjfzaOtRLLnbtvSGEgSe75Mfv1uCv7Pw7l3HKDu7yehNvfQdCpDq2KoVMnqjykIDDXL\nIpE+Rk7iJHakSnisBMIoOvzN6kpq5zo6NVsOFTvirQu0nXnk1RqUXVTb6zHZ9J0DVz7B6QqDs321\nZq5MoQvD68ysPsUwtw76Qm4sJFNHy7mFN/f+BXppZSRZGJ0K+qqgqUnNaBLNWqk52rM9VTPjM1zg\n4Pg8OHjrul6Nxya7nL+qCneTaQ2O+uCUaUH2lFfRjVj0ZFnkwjQkU6ti6C0k6yvhSK8EI2lK0iFh\nS0R0pPU3XiI+of1h4sKz2HoSNWBwxmaFC3dVEi4StE60+6cOJoFySdStMPqDNlP0wfws+yF0mgtC\n0d8M3WlIR3r5YaWaksRImppE2r3sXWE4BYYKoRKzf6otFNljW1v6v5ixiljFuyRShUCZxqnvhnvu\nPNYHqyIX6C9fRYzhYVnE6CcLA6xpScIK1Y6edt2r8NO1P7Srg/cD05+oRPULdt2fXhLc+nFbOKL7\ncTdG8R1rM3vjfscJWEWhs2xdjASrAjKwLIQQDiHEdiHEr63Hk4QQW4UQ+4UQvxBCOK3tLuvxfmv/\nWf0z9C7IxgfX3YXUYV+kwEmgzEXYp3DVM3/X93PbdKLgADibM3vOzavMz6K7vqbZ7jw23MlkGvLX\nwO6Ex98DfiSlnAKcAmJlse8BTlnbf2QdN7D0QjB6k5wTq6x058oX2Lq6Ju0Erfpoa8bnGqnMWlOF\nGpIoumTm1tvTes5TLaU0T5McvlHh9Fme9h2WUMSc18OJ/p6CQJpiIYQYB/wZ8Kj1WABXAhutQ54E\nvmzd/5L1GGv/VdbxA0tfLYyu/BcWUnNw5Bonp8c5+N5bf5bRS1/yP3+bVHfBpms0v8TZYuD0Sz6c\n80xaz7mroJ79t6/DXeugZaJCpMBsHmR4neilPk5dNDqjMXTnrxgpUxBI37JYC3yL9qpzJUCTlDL2\nDToKlFv3y4FPAaz9zdbxA0+G60tSZXh2hYhEmfR8AE+9+S+ZtaYq7T4jB29dR/1z4zln851pj22k\nMXfZEia/fjdFB8KowSjuxihzly3J6DVCY6MU7zEd0lGvSrDMw/6vesj/JEhg+hggYSoyhL/0A2FV\nQBpiIYS4ATghpdyWzRMLIRYJId4TQrwXlkOzT0OkwEmoSOFrc/6Aq0ly6Kb0y/A5/RLfS6YzNBNP\n/3AmsYO9oQrOekJB0Q3zFjHQ3aaBOqOmszBXvHwPs9ZUxad4M7fezsGb17N09c9omegknK/RXKEy\n6fkIanMIz8HGgXtj/chACQWkkZQlhPh34E5AB9xAAbAJ+FNgrJRSF0LMBVZKKf9UCPE76/4WIYQK\n1AKjZTcn6nNSVjqkGSVJ6fRKnOtaj2MVoaMeFUMz+2VuWbOOBQeu7rHXSMXL93DGi+Z5jl8fofAd\n14hf7h6rShYqljy04HF+2zyTj+4/N75fagq1czwYDvAdk2xdbfqHJj2/iKKdKi2XBRnzoouwT+Cf\nCL7DECwThEoN8o4pjLrqONr3i9FawsmNhGLVsbqwLHJ9CtJXscgkKSujDE4hxBeAZVLKG4QQzwLP\nSSl/LoRYB+yUUlYLIf4SmCmlXCKEuA34ipTyz7t73QERC+i9YMTEwus2/1qLlGIZnLpH5Y3HH6Hi\n2SVpZ2/OX7QIRZfoXoVj8wXfuuYFlhQdY+baKj5cWs2q+rNZUfpx+u9tiDL1qUqK9oL3RNTKUxG0\njlUY814rSsToVLLf0BSiHpWmyU4m3bmPw09Mxd1kxKt4t0x0ooYk3jrzi6x7HNTPVPEdk+huKDoQ\nThaMISwW2bAqMhGLviRlfRt4QAixH9Mn8Zi1/TGgxNr+ALC8x1eScmCSXvq6alVzoBe6zTUHCT1G\nYuRNaEl6HHNiJq5MPfehKr5TNws1aC6ndjbr3Pb5/2NJ0TG2tYXx1kouXlHJf//XlX0a61Bh3101\nRHwCRZe4G8J469rYsbwaRyASX2YeEwwRiaJYoVBvvcGhp6fSWi5onO5A0Q0cQZ1RHwfI/ySII9ju\nb1Ki4D2hU3QgzGtPP5iVU90AABM2SURBVMZvf/U0bWN9GIWdC/kmkstCMRjkxtoQR6m81HtD/LFw\n9nMn8x4sjK6mIge+Xk7JLvNiDfsEvs/MC9JQBf4zHWxb2R46PaL7ua7mW+y6v5rfBlxUvroQd0mQ\n4mfzcLa0110A0D1q/HUMpyBQqhC4xt9t9++hyNmPVPHxvZ2nWw82VvDCsqvMX3xLDJRA5y9qzKL7\n5IZ8oi5J0V7iGbPlbyZbIuY0UeXAnztxnFY466UQjoBO63gvvoOn0QtdaPUBRLO/S8sil8UiW76K\nIb82JPFD6hfh6CHTM9UydoBJm5rwV5hdud9dtY65y5aghiSGKgiVCM7ZfCehZhcXTv+EDzZPpaBB\nMnfZEkJFCo4Kgza3k8+u1Rn3ogOtxfpSRMyl77EFaToOwtc1sztFmPDsR6poKw9z6PpHs/N/GECe\n8xfgqUu974Hig/xKba821lVKfSxz1lMHpbvML3+kwInznWDS82KL/0REoWyzACTCEpL8PU0Q0dEi\nUQyvE0cXyV65LBSDRU5aFl2RdeHoRjBS+S2kpsargAfLPISKHbzx3f/g0geXErq4FeOYh8K9gkl3\n7mPn0XLKnzTHG/2bek4FPLSFNCauU8z04w5zcTDDe+F8jVCxA3955/Tmi1eYPVdbJkOkPMzBax8j\n15m9ugpPvYGzxVzWf/JC2cmvE/P1zFpTxZlvn44X2O2WrlK2487odgf08ctcRPIkUx8/0akUQTxc\nnkGbwsEWi2xGQAbKZzHgyHA4u76NXnzosTm083QE7wmdax74K+beth1x0EvhXoHTLzn5vQrGP6qa\nfTWDOo4flTLmYQ+RZlcnoYi9JkCoxEm4wPxItIQkz1hjZqffbKdYvEsmCcWMmsHrbxJbtAVwzuY7\nOX9V8lgcIYkaMiuNOf0Gz970n4C5fPya3Tey4MDV+A4rnP1IFf5ZbSnP0bEhkIjoiECo6/4fFlGP\nyuHrNDx1ksnP+Tsd01XW7kgRikwZUmIRI6ui0ZUXPI30b0M1/QuvbDkPZ7Pg8sXvsmXNOhS93VoT\nEQNXQwitJcz0x4KmFz7BcZfovKu9RKF5ssK3VvyUYFn7axQchMvvW2z6OgIGW9aYv8zrmsw8uPzD\n5rHT3lwImAuvfhtwcemOBQBZyRZ9sLEiqTBxTMBe2XJefFvbCS/eeiNuAYFZmv/o9VEU3cBTG+Jb\n36zkim/ci7vJoPZ34zl6uoi5t21n9AdRpv04nGRVdNkQKIGOIgJY5RKjHPpKuy9o4o/3d/0iI3Ra\nkSlDahrSHX2eoqSYksSnIgmhU6k5CI3Nw3+mua3gcJhwYbvrp2mKg6L9UZSwxNXYFne6pWrIm/ja\nYHZyj3pVzn3wQx4Y/XseOPzleM7G/EWLcJ6OxP0bJ8/zsGN5NbvDAZbctxRFl7zx+CNc8N5Xyfuv\nIuq/FsD7ig81ZDoBDRUiPkFwjMy4yPD0Jyrx1AnLSgA1ZF4zx+dLxr8k+dLqV3mg+CBXfONelIhB\nuFDlp2t/yAOHv0zDv01KclwamsLJ2XmILzbgVKOcOFGIZ48Lb52k9P1mRMC0LnqyGlKS8L+MTRkb\nZxWi6JLCfa0ozYF4fkXSj0GCWIw0q6Lf8iz6i2yIRYw+iUYHwegoFoHpY1CDOgfuVih/XsVTa154\nkQKnWW5PN6g/18P2FdVcft9i3A1h1Oa2roUikYREr+BYNyfPcxCe2MaYMc1oj5bgqQ0lfeFC/9RC\nbWMBe+c/yTVfM9v1xYrVxvq0GqpA9ypWQVtoLRcpoxHpcvGKSnNKEZIoYXNxlxIxMDQl/v5jYhYo\nc6HoEk9tCEcg+UumF7oI52sYTsHRa83rr+LZKM6GICLQ1lkoMllOnigYlriHS8zFZI6gjnak3tyX\nwleRy5Ww+mv6MaLFIkavRSOVYFjOzcgZBTROd1N0IGxe2JZ3vv6CQkZ9HIh/cYJj3TSf5aBsW7Bd\nLNK44KXXRbjEw4kLXZTujKAGdVommt3Rxr96OmWCUqfXSCjzlygWdy9/ntHqaW7xtXR6TibMWV6J\nuzEaF4rEQsZKwnLwY1fkU/7G6ZQl+WMh0GCZ+SX21AW77hrWGxIybqXXRdSrESlw4vm0pT1UGsMS\ngFwWCsgNsRiSPot06LVfo4uLQi/1ISIGBUeSK2aJSJTS95tRm0MogTCOQAQ1YFkAqkLU20F8Us2x\n4/uiGJrCmG1tuGtbUZvbKNwf6CQUsWMdgUhSEWFze/taCjDzQbaurmFB/t4+CwXA1tU1GE7z2oq1\nRAASzm+Kw7iXm+L+GaBTtTERMTM2/Wc6OgtFh9aCfaV5ihd3bSt6oTt5xxDxVQymUzORYWtZpCJj\nayPByki0MKTXhdLcuSZFbO2IXuoj6lHRWswvS6pju6Lb0m8p9sXCrrG/iUWEj17pZM/dNTzYWMED\nxQfTHkO6XHXnPTiCeqe07LSmXRAPf3YMZ8bIpMZIV6FuvdSH1BScRxqTu6SP8OlHjCGflNVfxC6K\ntEUjIXlLRiIIiIftkogX9LXaCEQM9n1DcM73Az168zuSeHwn4YjtSyr7F40nISWW+jM0hYhPcumO\nBfxh9kayybkPVVFw2MCtRVF0BRKmH90JRZe+iBTHZtzTo2PXuYhOZEIx2vEW87y9FAqbdkaUWMTI\nSDQ6CkaqVPDEi11TcdS3MOPfUrxWOsLRVf3Pjq/TjQXSNNmNu8nAd1hhysX1PZ8zQ2LJYrPWVFHy\nEfEmxF3RpWBmQSS6e77zwInkbRkkXuUCuTL9iDEixSJG2qKR0AWt48XcSTz6OtfuIDzQhWikEAwR\niYKmUPQXR/n/7Z1bbBzVGcd/Z2Z2vbdsHNvBDjYE2iJoWu4pCi1QCmpFL1KLmlaqekEIKVJAbfPA\nQx6oVFVU4oEHVKkuzQuC0qIiWlRaWiACUVEKUkMxCaQ1JCkJdrCDnfi63t2ZndOHmVmv13uZ2Ys9\ns56ftNrdyVzOeGf++c6Z7/y/428Pglbg8Ytebq49Ffje+zdz+KkdLFyRIzkRoWu65Pgl57CWIlGJ\nRmeTrlxvfbogfhMK2OBi4eBJNMqeljTVr65HLafxCv/udEOmFxMkxxXMNpX6fPyil9mR3GFVkz9r\nWI9OoWzcYh2Fos4NHgpFY3Ts05BGcHURNVk20Xm5xk3fv4xNBzaTmJDoV7TPGPjo3mGe/fmDTO+I\nkOvpYv7ipCVWHpzRoQVC4Vgnlr5q4Peuh58JI4syXEUZTRZnhgoDcrWoNUahG2DnLRQS1jpnd2gr\nCga3i+t/ey9Kr2ThfJWe0TrWiK0SimbE2oNQhFHFanwRWUgp1z3ppRxXeRoeDYErHsdtpFHHbXzi\n+k0cfOIRMv1dRG6YLs7daBeXP3Q3PUehexQUAz64JV7MKWkqA7McD1FDLUKhaB5fiIWD3wQD1lY0\nvFJ6Uw68Ns9n7tvL/JCCfK6X1Kn2Vl84sm8YLSuJnS0QXZBc8NISY7dWcJ6qIhR1z7cFf9PisQLQ\n9fC7UIDPxAL8KRjQ/vEMcHEDVbnxnIzI9Mk88Slr7kZsxlw1XbxVfPLV73PVA9a+FUOSHFtCzRhs\n//OMq+1rnmeLRcKrUPj1+vMDvhMLsH4wP/5onqKMJgo1NxZlFFCXDNIns6ROG0TnTHJtqtbyn8/9\nhpH9w5y7VMFIKMXjLztVVR8Kq3puLRQJCEY04RCEqAJ8KhYOfhQM8HAhNikaFanR/7dMbq15Idke\nlegs7BrZ3bZyiUvb82z+0SmWBmLFyWG12lpTKFpEM14n63G9BUUowOdiAf4WjHaLRiMRhjOhKzMg\nGNk/zOtXPUWfWtvF2guOoc5NR24HQ2HsjxcTnTXsOSmqtxqiPoomQqGoj+/FAvwrGODxAm3g5mhE\nMLK9EY7sG25LPVVnnsncs9tInNQY2T/M2C1RZEQh1xsjN5CqPOGtDRmaK/YfoG4HBE8owCezTtNK\nr9wVua3ueqKJvIa1oCEPjUYKH5VVSJOJLvu7WvTTmB9SmbsmB4saJ27/NWA5Xo3e6c0lq5TLH7qb\n1JgVuURtd/J82vLNWBgUy25XeqFYwMehmjNVs7RCJDZyVNGxfhZ+jjCgwQu3kXMqmz9SSERWVkiL\nCdJf/ZChbWfZ8pbCrpHdXH3/3Vx+03vej4Xlvg0QnbEqmicmdbQlywDH1ARaVvLOD4fpPp6taPTj\nZ6FYD/wiFF4JlFhABwtGvTTlsjBeRrSiFd/SQIxcb8yqu2pb641/1M3koQESUyaTp3qIzZiM/+oT\nZMw8D88M8vGX7qx5PMcM+GMv3EX6uMK1P92LnhLk0yuLFWtZiZYx+eJ37rRmn5aZ+rRlglgLDZvX\n+noKqlBAwLoh5XRkt8Shyrk53RGZiGH0pVgcipOYzGHENc5cHSGyCEv9kuisoPtYgeisgWKYVgGj\nuMpHV2skx63fXE8JCjFY3F4odlVKcXwrHIPej65UufD5xRX2eYWExsS+HLzWTc9Rg+S700X/CGht\nVNHqSGIthcKvIrFhzG9kXve1YMh8vnHBqFI1zfHUELqBNrXAJtulSs0YRGcjdJ/Q2fIuZM7TyHYr\nxCdsQx7ASETYMmpiataYQz6rFMsKVGzCZokRE0TnTKLzOhc+b65yxVIzoB3sZuvIIoW45SImZmvX\n5WiEIAtFpxC4bkg5fv/Rm7rIa9U0scN9dWoOkcmhzGYY+OcMsYnl//mdWqyKbqJmDDb9b5HEZI7E\npF4cb7jiwepZnuZQlsVByyfT2U+p96jQCyiZPNv+Po2a0cn0R6qWHlyreR1+xK9RhVcCLxawAQSj\n2vmVmcxYNvoFjLhKdMFKzCqtW2K9rGrjTmFmqcH9U5dV3P3g1hk2HzdJna5cQa1orKsbiEyOLW+d\nK35vRVTR8gp0xf2G3Y9GCHQ3pJSO7pLAqm5J0eLPnr5eaoyzcL7K1jcXVlUilxG1WK4AIHa2gBFT\nua/vvyvWeyOX57uP7UNqkuiQIDYlyJynMn2l5LID51Y87Vhl3V8pw7SRhLQ2RROhUDROx4gFLF8I\nfhWNtgoGWKKRydL/6tkq9nyWqW8hrmHEVfJphXOfzfOHhXRx1W+m5ri2K0pyXBKbkSiGyeS3smx/\nWKHv38u1SGtVDGvaRzMUCl8S6KchtfCrYEDrSy1Ws8FfRURD70tw8IlHuPSRvfS8Y/32WtYayPz9\nAw9y80s/RusySL6SpPuEXixHmE8p9ByerVwxDFqSgNXusYm1EosgCUXgKpK1Qyxg4whGVbetstqf\nzjJn0tfpGzfRe9SqfGZqCvnNGvNDKqZGcbkoKWBUsWoYVBcKcC0WnSAUQRIJhw3z6LQefu6WtLJL\nUrNEQblDONj1TwoMvL48G1UBlLwksiDJdVtlD7UlVj0qrXiMJukEodgIuHoaIoR4XwhxRAgxIoQ4\nZC/rEUIcFEK8Z79vsZcLIcQvhBDHhBCHhRDXtPMEgkwrbxK34wSlCVPabLZYAlHoJqe+ZhVPvnfP\nkxgxUbZdhXqtTVQQK27TIUIRxKjCK14enX5BSnmVlHKn/X0/8KKU8hLgRfs7wJeBS+zXHqDxmUst\nwq9mOk1TXjSn0s3qwpJf6AUU3WToBcGWUZOf/WV3caJYzajCY/vKCYUiWDSTZ/F14FH786PAN0qW\nPyYtXge6hRDbmjhOy/CjaDSdS9CAYIgyoXDe4xNZYtM68TMK0XnLSGeVoU0VvEQV7cqfWHmMUCha\njVuxkMALQog3hBB77GX9UsoP7c8TQL/9eRD4oGTbMXtZSEBIjZkY8dXFizwZ21RhLbIx12owcyMJ\nBbgXixuklNdgdTHuEULcVPqP0nqk4umxihBijxDikBDikC7r1JxoMX6NMBrGTXRR7/gRFT0dxYhb\nwnD6Ro2dv3wTPR1ddsEqrutCNCr8fTtJKDYirsRCSjluv58BngauAyad7oX9fsZefRy4oGTzIXtZ\n+T4PSCl3Sil3RkSs8TNogo4SjPJ9lQtGjdyIQiKCGbHMdzPnqaTGc2x/NsOrP9nFwmCU2w68Yq2T\niGL0pZbzOFoQabSSUCjaS12xEEIkhRCbnM/Al4C3gWeAO+zV7gD+ZH9+BviB/VRkFzBb0l3xHX6L\nMhoWDC/JT2X1UfV0lKWBGEZM0H08i5oxUHST+OQSm49l+Ns9n0dPR9HTURaH4piJKPOf7rN24FIw\nOmGMYiMLBbjLs+gHnhZCOOv/Tkr5nBDiX8CTQoi7gJPAt+31/wp8BTgGZIDaLis+we9zSxqhav6F\ng32jK7rJxHUaQy8bqBljha2/sBO4jLhGtjfC9KcUIEn63TlkyZyUFZR3iwIuFBtdJBzqioWU8gRw\nZYXl08CtFZZL4J6WtG6N8YtgNJ2wVWvfZVGFMxZx/j8KyzNUMyVzQOz32MQiXdMqyTGFQtyy8tP0\ngpXkVSMxK+hCEbKML9K9hRDzwOh6t8MDfcDUejfCJUFqK4TtbSeV2rpdSrnVzcZ+GaEaLUn28j1C\niENBaW+Q2gphe9tJs23tCPObkJCQ9hOKRUhIiCv8IhYH1rsBHglSe4PUVgjb206aaqsvBjhDQkL8\nj18ii5CQEJ8TikVISIgrQrEICQlxRSgWISEhrgjFIiQkxBX/B1ixHSMoC1KdAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "\u003cFigure size 288x288 with 1 Axes\u003e"
            ]
          },
          "metadata": {
            "tags": []
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "#@title Solution: Mandlebrot set (Double-click to reveal)\n",
        "\n",
        "MAX_ITERATIONS = 64\n",
        "NUM_PIXELS = 512\n",
        "\n",
        "def GenerateGrid(nX, nY, bottom_left=(-1.0, -1.0), top_right=(1.0, 1.0)):\n",
        "  \"\"\"Generates a complex matrix of shape [nX, nY].\n",
        "  \n",
        "  Generates an evenly spaced grid of complex numbers spanning the rectangle \n",
        "  between the supplied diagonal points. \n",
        "  \n",
        "  Args:\n",
        "    nX: A positive integer. The number of points in the horizontal direction.\n",
        "    nY: A positive integer. The number of points in the vertical direction.\n",
        "    bottom_left: The coordinates of the bottom left corner of the rectangle to\n",
        "      cover.\n",
        "    top_right: The coordinates of the top right corner of the rectangle to\n",
        "      cover.\n",
        "\n",
        "  Returns:\n",
        "    A constant tensor of type complex64 and shape [nX, nY].\n",
        "  \"\"\"\n",
        "  x = tf.linspace(bottom_left[0], top_right[0], nX)\n",
        "  y = tf.linspace(bottom_left[1], top_right[1], nY)\n",
        "  real, imag = tf.meshgrid(x, y)\n",
        "  return tf.cast(tf.complex(real, imag), tf.complex128)\n",
        "\n",
        "c_values = GenerateGrid(NUM_PIXELS, NUM_PIXELS)\n",
        "initial_Z_values = tf.zeros_like(c_values, dtype=tf.complex128)\n",
        "initial_diverged_after = tf.ones_like(c_values, dtype=tf.int32) * MAX_ITERATIONS\n",
        "\n",
        "# You need to put the various values you want to change inside the loop here\n",
        "loop_vars = (0, initial_Z_values, initial_diverged_after)\n",
        "\n",
        "# this needs to take the same number of arguments as loop_vars contains and\n",
        "# return a tuple of equal size with the next iteration's values\n",
        "def body(iteration_count, Z_values, diverged_after):\n",
        "  # a matrix of bools showing all the co-ordinatesthat haven't diverged yet\n",
        "  not_diverged = tf.equal(diverged_after, MAX_ITERATIONS)\n",
        "  # a list of the indices in not_diverged that are true\n",
        "  not_diverged_indices = tf.where_v2(not_diverged)\n",
        "\n",
        "  # Gather the values for just the undiverged co-ordinates, and generate the \n",
        "  # next iteration's values\n",
        "  not_diverged_c_values_array = tf.gather_nd(c_values, not_diverged_indices)\n",
        "  not_diverged_Z_values_array = tf.gather_nd(Z_values, not_diverged_indices)\n",
        "  new_Z_values_array = (not_diverged_Z_values_array * not_diverged_Z_values_array\n",
        "                        + not_diverged_c_values_array)\n",
        "  \n",
        "  # merge the new values with the already-diverged\n",
        "  new_Z_values_or_zeroes = tf.scatter_nd(\n",
        "      not_diverged_indices, \n",
        "      new_Z_values_array, \n",
        "      tf.shape(Z_values, out_type=tf.dtypes.int64))\n",
        "  new_Z_values = tf.where_v2(not_diverged, new_Z_values_or_zeroes, Z_values)\n",
        "\n",
        "  # And now we're back to the original code\n",
        "  has_diverged = tf.abs(new_Z_values) \u003e 2.0\n",
        "  new_diverged_after = tf.minimum(diverged_after, tf.where_v2(\n",
        "      has_diverged, iteration_count, MAX_ITERATIONS))\n",
        "  return (iteration_count+1, new_Z_values, new_diverged_after)\n",
        "\n",
        "# this just needs to take the same number of arguments as loop_vars contains and\n",
        "# return true (we'll use maximum_iterations to exit the loop)\n",
        "def cond(iteration_count, Z_values, diverged_after):\n",
        "  return True\n",
        "\n",
        "loop = tf.compat.v1.while_loop(\n",
        "    loop_vars=loop_vars, \n",
        "    body = body, \n",
        "    cond = cond, \n",
        "    maximum_iterations=MAX_ITERATIONS)\n",
        "\n",
        "with tf.Session() as sess:\n",
        "  results = sess.run(loop)\n",
        "\n",
        "## extract the final value of diverged_after from the tuple\n",
        "final_diverged_after = results[-1]\n",
        "plt.matshow(final_diverged_after)\n",
        "pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "kAlaMKj0gZdX"
      },
      "source": [
        "## SparseTensor\n",
        "\n",
        "[Full documentation](https://www.tensorflow.org/api_docs/python/tf/sparse/SparseTensor])\n",
        "\n",
        "A sparse tensor is created from a list of indices, a list of values and a shape: the same as the arguments to scatter_nd. Any element within the tensor that doesn't have an explicit value will be treated as zero. So a sparse tensor can be viewed as a deferred call to scatter_nd.\n",
        "\n",
        "For large tensors where most of the values are zero, sparse tensors can grant major savings in memory. The [tf.sparse module](https://www.tensorflow.org/api_docs/python/tf/sparse) contains several specialised operations that know how to work with sparse tensor's internals and skipping all the zero values, thus granting major savings in processing speed as well.\n",
        "\n",
        "Similarly sparse tensors can be efficiently divided or multiplied by a tensor or scalar. But attempts to perform inefficient operations on a sparse tensor (i.e. ones likely to set most elements to a non-zero value) are not allowed. You need to convert the sparse tensor to a normal, or \"dense\", tensor with the ```tf.sparse.to_dense``` function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 840
        },
        "colab_type": "code",
        "executionInfo": {
          "elapsed": 447,
          "status": "ok",
          "timestamp": 1568714727763,
          "user": {
            "displayName": "Mark Charsley",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mAuhiLGs9-CmN1BMtie2tWn3oucS__w2c4gNC6yBg=s64",
            "userId": "18064761308611293318"
          },
          "user_tz": -60
        },
        "id": "qDJmkGN2lBhB",
        "outputId": "c53c8ed7-a5bb-4b6c-e357-bf65bc28605b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "sparse =\n",
            "SparseTensorValue(indices=array([[0, 0, 0],\n",
            "       [0, 1, 2],\n",
            "       [0, 2, 1],\n",
            "       [1, 0, 2],\n",
            "       [1, 1, 1],\n",
            "       [1, 2, 0]]), values=array([111, 123, 132, 213, 222, 231], dtype=int32), dense_shape=array([2, 3, 3]))\n",
            "\n",
            "sparse * dense =\n",
            "SparseTensorValue(indices=array([[0, 0, 0],\n",
            "       [0, 1, 2],\n",
            "       [0, 2, 1],\n",
            "       [1, 0, 2],\n",
            "       [1, 1, 1],\n",
            "       [1, 2, 0]]), values=array([12321, 15129, 17424, 45369, 49284, 53361], dtype=int32), dense_shape=array([2, 3, 3]))\n",
            "\n",
            "dense * sparse\n",
            "SparseTensorValue(indices=array([[0, 0, 0],\n",
            "       [0, 1, 2],\n",
            "       [0, 2, 1],\n",
            "       [1, 0, 2],\n",
            "       [1, 1, 1],\n",
            "       [1, 2, 0]]), values=array([12321, 15129, 17424, 45369, 49284, 53361], dtype=int32), dense_shape=array([2, 3, 3]))\n",
            "\n",
            "sparse / dense\n",
            "SparseTensorValue(indices=array([[0, 0, 0],\n",
            "       [0, 1, 2],\n",
            "       [0, 2, 1],\n",
            "       [1, 0, 2],\n",
            "       [1, 1, 1],\n",
            "       [1, 2, 0]]), values=array([1., 1., 1., 1., 1., 1.]), dense_shape=array([2, 3, 3]))\n",
            "\n",
            "to_dense gives\n",
            "[[[111   0   0]\n",
            "  [  0   0 123]\n",
            "  [  0 132   0]]\n",
            "\n",
            " [[  0   0 213]\n",
            "  [  0 222   0]\n",
            "  [231   0   0]]]\n",
            "\n",
            "scatter_nd gives\n",
            "[[[111   0   0]\n",
            "  [  0   0 123]\n",
            "  [  0 132   0]]\n",
            "\n",
            " [[  0   0 213]\n",
            "  [  0 222   0]\n",
            "  [231   0   0]]]\n"
          ]
        }
      ],
      "source": [
        "source = tf.constant([[[111,112,113], [121,122,123], [131,132,133]], \n",
        "                      [[211,212,213], [221,222,223], [231,232,233]]])\n",
        "# create a list of indices where is_divisible_by_three is true\n",
        "indices = tf.where_v2(tf.equal(0, source % 3))\n",
        "# extract a matching list of values\n",
        "values_divisible_by_three = tf.gather_nd(source, indices)\n",
        "\n",
        "with tf.Session() as sess:\n",
        "\n",
        "  sparse = tf.sparse.SparseTensor(\n",
        "      indices, \n",
        "      values_divisible_by_three, \n",
        "      tf.shape(source, out_type=tf.dtypes.int64))\n",
        "  print (\"sparse =\")\n",
        "  print(sess.run(sparse))\n",
        "  # We can efficiently multiply sparse by a dense tensor\n",
        "  print (\"\\nsparse * dense =\")\n",
        "  print(sess.run(sparse * source))\n",
        "  # We can efficiently multiply a dense tensor by a sparse\n",
        "  print (\"\\ndense * sparse\")\n",
        "  print(sess.run(source * sparse))\n",
        "  # We can efficiently divide sparse by a dense tensor\n",
        "  print (\"\\nsparse / dense\")\n",
        "  print(sess.run(sparse / source))\n",
        "\n",
        "  # But attempts to perform inefficient operations on a sparse tensor (i.e. ones\n",
        "  # likely to set most elements to a non-zero value) are not allowed.\n",
        "  # You need to convert the sparse tensor into a dense tensor first.\n",
        "  try:\n",
        "    not_allowed = sparse + source\n",
        "  except TypeError:\n",
        "    pass\n",
        "  \n",
        "  # Running to_dense is exactly the same as calling scatter_nd:\n",
        "  print (\"\\nto_dense gives\")\n",
        "  print(sess.run(tf.sparse.to_dense(sparse)))\n",
        "  print (\"\\nscatter_nd gives\")\n",
        "  print(sess.run(tf.scatter_nd(sparse.indices, sparse.values, sparse.dense_shape)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Afi0jfqkI9as"
      },
      "source": [
        "# Functional ops"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "sAVQxV71JF59"
      },
      "source": [
        "## tf.foldl and tf.foldr\n",
        "\n",
        "[Full documentation](https://www.tensorflow.org/api_docs/python/tf/foldl)\n",
        "\n",
        "These two functions split a given tensor across its first dimension. The resulting subtensors are then each passed to an op along with an \"accumulator\". \n",
        "For most iterations, the value of the accumulator will be the result of the previous iteration. But for the first iteration, the accumulator will either be passed an initial value passed into the foldl/foldr call, or the first subtensor. The final iteration's result then becomes the overall result of the op.\n",
        "\n",
        "So the rough pseudo code of ```x = tf.foldl(op, [[1,1], [2,2], [3,3]], initializer)``` would be\n",
        "``` python\n",
        "result_iteration1 = op(initializer, [1,1])\n",
        "result_iteration2 = op(result_iteration1, [2,2])\n",
        "result iteration3 = op(result_iteration2, [3,3])\n",
        "x = result_iteration3\n",
        "```\n",
        "Whereas the rough pseudo-code of  ```x = tf.foldl(op, [[1,1], [2,2], [3,3])``` (i.e. no initializer supplied) would be\n",
        "``` python\n",
        "result_iteration1 = op([1,1]], [2,2])\n",
        "result_iteration2 = op(result_iteration1, [3,3])\n",
        "x = result_iteration3\n",
        "```\n",
        "\n",
        "```foldr``` is identical to ```foldl```, except that the order the tensor is iterated through is reversed. So the rough pseudo code of ```x = tf.foldr(op, [[1,1], [2,2], [3,3], initializer)``` would be\n",
        "``` python\n",
        "result_iteration1 = op(initializer, [3,3])\n",
        "result_iteration2 = op(result_iteration1, [2,2])\n",
        "result iteration3 = op(result_iteration2, [1,1])\n",
        "x = result_iteration3\n",
        "```\n",
        "\n",
        "The only complication of this method is that the op is defined by a python callable. Note that the callable is only called once, at execution time, to build the operation. **Your python callable is not called for every row in the input**, nor can it see the individual values. It is the op created by your python code that will be repeatedly called. \n",
        "\n",
        "Note that despite this, use of these methods still eliminates several optimisation opportunities that are present in tensorflow built-in operations. So if you can use something like tf.math.reduce_sum instead of these ops then your code may well run significantly faster."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 168
        },
        "colab_type": "code",
        "executionInfo": {
          "elapsed": 440,
          "status": "ok",
          "timestamp": 1568711374271,
          "user": {
            "displayName": "Mark Charsley",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mAuhiLGs9-CmN1BMtie2tWn3oucS__w2c4gNC6yBg=s64",
            "userId": "18064761308611293318"
          },
          "user_tz": -60
        },
        "id": "40cDr4ogOk8W",
        "outputId": "397c3127-6d7b-4050-93bb-d102df4fe4c1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Executing foldl\n",
            "In my_function.\n",
            "foldl result:\n",
            "[ 9 12]\n",
            "\n",
            "Executing foldr\n",
            "In my_function.\n",
            "foldr result:\n",
            "[20 24]\n"
          ]
        }
      ],
      "source": [
        "source = tf.constant([[1,2],[3,4],[5,6]])\n",
        "\n",
        "# element \n",
        "def my_function(previous_iterations_result, element):\n",
        "  print(\"In my_function.\")\n",
        "  # this depends on the previous values, thus highlighting the difference \n",
        "  # between foldl and foldr\n",
        "  return tf.math.maximum(\n",
        "      previous_iterations_result, element) + previous_iterations_result\n",
        "\n",
        "with tf.Session() as sess:\n",
        "  print(\"Executing foldl\")\n",
        "  print(\"foldl result:\\n%s\"%sess.run(\n",
        "      tf.foldl(my_function, source)))\n",
        "\n",
        "  print(\"\\nExecuting foldr\")\n",
        "  print(\"foldr result:\\n%s\"%sess.run(\n",
        "      tf.foldr(my_function, source)))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "P9xLy92GtXHx"
      },
      "source": [
        "## tf.map_fn\n",
        "\n",
        "This op is similar to foldl, but the python function only takes a single argument, it lacks the accumulator argument containing the result of the previous iteration. Again the callable is called just once, and is used to generate an tensorflow op. It is this generated op that is executed once per row. And again, be aware that replacing the map_fn call with a built-in op - if possible - can result in significant increases in speed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "height": 69
        },
        "colab_type": "code",
        "executionInfo": {
          "elapsed": 465,
          "status": "ok",
          "timestamp": 1568886060131,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          },
          "user_tz": -60
        },
        "id": "x4CPdY9bzhjs",
        "outputId": "d6c582fa-a8e5-49ea-bb7f-bd0d2c67b819"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "In my_function\n",
            "foldr result:\n",
            "[ 3  7 11]\n"
          ]
        }
      ],
      "source": [
        "source = tf.constant([[1,2],[3,4],[5,6]])\n",
        "\n",
        "# element \n",
        "def my_function(element):\n",
        "  print(\"In my_function\")\n",
        "  \n",
        "  return tf.math.reduce_sum(element)\n",
        "\n",
        "\n",
        "with tf.Session() as sess:\n",
        "  print(\"foldr result:\\n%s\"%sess.run(\n",
        "      tf.map_fn(my_function, source)))"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "Introduction_to_TensorFlow_Part_3_-_Advanced_Tensor_Manipulation.ipynb",
      "provenance": [
        {
          "file_id": "11mikyGawc9mOOVIXZENnm7TyFyyDyByB",
          "timestamp": 1568382156239
        }
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 2",
      "name": "python2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
